Showing preview only (304K chars total). Download the full file or copy to clipboard to get everything.
Repository: microsoft/types-publisher
Branch: master
Commit: 820f454e08b0
Files: 70
Total size: 285.3 KB
Directory structure:
gitextract_h4stda0m/
├── .editorconfig
├── .gitignore
├── .travis.yml
├── .vscode/
│ └── launch.json
├── CODEOWNERS
├── DEBUGGING.md
├── IISNode.yml
├── LICENSE
├── README.md
├── dependenciesWhitelist.txt
├── jest.config.js
├── package.json
├── server.js
├── src/
│ ├── .vscode/
│ │ └── launch.json
│ ├── calculate-versions.ts
│ ├── check-parse-results.ts
│ ├── clean.ts
│ ├── crawl-npm.ts
│ ├── create-search-index.ts
│ ├── full.ts
│ ├── generate-packages.test.ts
│ ├── generate-packages.ts
│ ├── get-definitely-typed.test.ts
│ ├── get-definitely-typed.ts
│ ├── index.ts
│ ├── lib/
│ │ ├── azure-container.ts
│ │ ├── blob-uploader.ts
│ │ ├── common.ts
│ │ ├── definition-parser-worker.ts
│ │ ├── definition-parser.test.ts
│ │ ├── definition-parser.ts
│ │ ├── module-info.test.ts
│ │ ├── module-info.ts
│ │ ├── npm-client.ts
│ │ ├── package-publisher.ts
│ │ ├── packages.test.ts
│ │ ├── packages.ts
│ │ ├── rolling-logs.ts
│ │ ├── secrets.ts
│ │ ├── settings.ts
│ │ ├── versions.test.ts
│ │ ├── versions.ts
│ │ └── webhook-server.ts
│ ├── make-server-run.ts
│ ├── mocks.ts
│ ├── parse-definitions.test.ts
│ ├── parse-definitions.ts
│ ├── publish-packages.ts
│ ├── publish-registry.ts
│ ├── test-get-secrets.ts
│ ├── tester/
│ │ ├── get-affected-packages.test.ts
│ │ ├── get-affected-packages.ts
│ │ ├── test-runner.test.ts
│ │ ├── test-runner.ts
│ │ └── test.ts
│ ├── types/
│ │ ├── adal-node.d.ts
│ │ ├── azure-keyvault.d.ts
│ │ ├── fstream.d.ts
│ │ └── npm-registry-client.d.ts
│ ├── upload-blobs.ts
│ ├── util/
│ │ ├── io.ts
│ │ ├── logging.ts
│ │ ├── progress.ts
│ │ ├── test.ts
│ │ ├── tgz.ts
│ │ └── util.ts
│ ├── validate.ts
│ └── webhook.ts
├── tsconfig.json
└── tslint.json
================================================
FILE CONTENTS
================================================
================================================
FILE: .editorconfig
================================================
root = true
[*]
charset = utf-8
end_of_line = lf
indent_size = 4
indent_style = space
insert_final_newline = true
trim_trailing_whitespace = true
================================================
FILE: .gitignore
================================================
bin/
cache/
data/
logs/
node_modules/
output/
validateOutput/
\#*
.\#*
================================================
FILE: .travis.yml
================================================
language: node_js
node_js:
- 'node'
sudo: false
branches:
only:
- master
install:
- npm ci
git:
depth: 1
================================================
FILE: .vscode/launch.json
================================================
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "parse-definitions.js",
"cwd": "${workspaceFolder}",
"program": "${workspaceFolder}/bin/parse-definitions.js",
"args": [],
"sourceMaps": true
},
{
"type": "node",
"request": "launch",
"name": "calculate-versions.js",
"cwd": "${workspaceFolder}",
"program": "${workspaceFolder}/bin/calculate-versions.js",
"args": [],
"sourceMaps": true
},
{
"type": "node",
"request": "launch",
"name": "publish-packages.js --dry",
"cwd": "${workspaceFolder}",
"program": "${workspaceFolder}/bin/publish-packages.js",
"args": ["--dry"],
"sourceMaps": true
}
]
}
================================================
FILE: CODEOWNERS
================================================
* @sandersn @RyanCavanaugh
================================================
FILE: DEBUGGING.md
================================================
# DEBUGGING
Debugging an Azure App Service has been a miserable learning
experience for me, so here are some tips to help.
First, repro and debug anything you can locally. Everything except the
webhook-server either includes a dry-run mode or runs identically
locally. But you may still need to test the non-dry-run code.
Your basic workflow looks like this:
1. `npm run push-production`
2. Resend webhook or merge a new PR.
3. Look at ftp logs.
(2) and (3) have some complexity.
Step (2) is required because an App Service doesn't seem to properly
restart until a web request comes in. If you're just testing startup,
or something else that happens even if no PRs need to be merged,
resending a webhook should be fine. Types-publisher only cares that
the push was to master and not to some other branch. This is true for
all non-team-member PRs.
Otherwise, you'll need to find a easy PR to merge.
For step (3), proceed to the FTP logs. If you can't remember the
address, you can find it in the Diagnostics logs section of the
TypesPublisher App Service page on Azure. You can also set up new
username/password combinations somewhere, but I don't remember where
exactly. I think the README may tell where.
Then look in LogFiles/Application/ and page down to index.html, which
contains a *sorted* list of log files. You'll see a new log file
every time the server restarts, which is at least every time you push
to production, but may be more often if the server is crashing.
================================================
FILE: IISNode.yml
================================================
# For all options see https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/iisnode.yml
nodeProcessCommandLine: node.exe --stack-trace-limit=1000
loggingEnabled: true
devErrorsEnabled: false
maxLogFileSizeInKB: 2048
maxTotalLogFileSizeInKB: 65536
maxLogFiles: 16384
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) Microsoft Corporation. All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE
================================================
FILE: README.md
================================================
# This repo has moved.
This service is now part of [microsoft/DefinitelyTyped-tools](https://github.com/microsoft/DefinitelyTyped-tools/tree/master/packages/publisher).
If you need to update allowedPackageJsonDependencies.txt (formerly named dependenciesWhitelist.txt), please do so at its [new location](https://github.com/microsoft/DefinitelyTyped-tools/blob/master/packages/definitions-parser/allowedPackageJsonDependencies.txt).
================================================
FILE: dependenciesWhitelist.txt
================================================
@babel/code-frame
@babel/core
@babel/generator
@babel/parser
@babel/template
@babel/traverse
@babel/types
@elastic/elasticsearch
@electron/get
@emotion/core
@emotion/serialize
@emotion/styled
@hapi/boom
@hapi/iron
@hapi/wreck
@jest/environment
@jest/fake-timers
@jest/globals
@jest/transform
@jest/types
@loadable/component
@material-ui/core
@material-ui/types
@sentry/browser
@storybook/addons
@storybook/react
@types/base-x
@types/expect
@types/firebase
@types/highcharts
@types/hoist-non-react-statics
@types/mkdirp-promise
@types/next-redux-wrapper
@types/ink
@types/js-data
@types/react-native-tab-view
@types/react-navigation
@types/three
@types/rsmq
@types/vue
@types/webdriverio
@types/winston
@types/wonder-commonlib
@types/wonder-frp
@uirouter/angularjs
@vue/test-utils
@wordpress/element
abort-controller
actions-on-google
activex-helpers
adal-node
ajv
algoliasearch
algoliasearch-helper
anydb-sql
aphrodite
apollo-client
apollo-link
apollo-link-http-common
ast-types
aws-sdk
axe-core
axios
base-x
bignumber.js
bookshelf
broadcast-channel
cac
cash-dom
cassandra-driver
chalk
chokidar
cids
commander
connect-mongo
cordova-plugin-camera
cordova-plugin-file
cordova-plugin-file-transfer
cosmiconfig
cropperjs
csstype
cypress
date-fns
decimal.js
del
dexie
dnd-core
dotenv
egg
electron
electron-notarize
electron-osx-sign
electron-store
ethers
eventemitter2
eventemitter3
expect
express-graphql
express-rate-limit
fast-glob
fastify
final-form
flatpickr
form-data
gl-matrix
glaze
globby
google-auth-library
google-gax
got
graphql
graphql-tools
grpc
handlebars
hoist-non-react-statics
htmlparser2
i18next
immutable
indefinite-observable
inversify
jest-cucumber
jest-diff
jest-environment-node
jest-environment-jsdom
jest-mock
jest-snapshot
jimp
jointjs
levelup
lit-element
localforage
log4js
logform
loglevel
logrocket
knex
knockout
magic-string
mali
meteor-typings
moment
monaco-editor
moment-range
mqtt
next
nock
normalize-url
objection
opentracing
parchment
parse5
path-to-regexp
pkcs11js
popper.js
postcss
polished
preact
pretty-format
probot
prom-client
protobufjs
protractor
query-string
quill-delta
raven-js
re-resizable
react-autosize-textarea
react-dnd
react-dnd-touch-backend
react-intl
react-native-fs
react-native-maps
react-native-modal
react-native-svg
react-native-tab-view
react-navigation
recast
redux
redux-observable
redux-persist
redux-saga
redux-thunk
rollup
rrule
rxjs
safe-buffer
should
smooth-scrollbar
scroll-into-view-if-needed
source-map
styled-components
sw-toolbox
swagger-parser
terser
three
treat
tslint
ts-toolbelt
tweetnacl
typescript
typescript-tuple
utility-types
vega-typings
vfile
vfile-message
vue
vue-router
vuex
webpack-chain
winston
winston-transport
xmlbuilder
xpath
zipkin
zipkin-transport-http
================================================
FILE: jest.config.js
================================================
module.exports = {
"roots": [
"src"
],
"preset": "ts-jest",
"testRegex": "(/__tests__/.*|\\.(test|spec))\\.tsx?$",
"moduleFileExtensions": [
"ts",
"tsx",
"js",
"jsx",
"json",
"node"
],
}
================================================
FILE: package.json
================================================
{
"private": true,
"name": "types-publisher",
"version": "0.0.0",
"main": "./bin/index.js",
"types": "./bin/index.d.ts",
"description": "Publish DefinitelyTyped definitions to NPM",
"dependencies": {
"@octokit/rest": "^16.1.0",
"@types/tar-stream": "^1.6.0",
"adal-node": "^0.1.22",
"applicationinsights": "^1.0.7",
"azure-keyvault": "^3.0.4",
"azure-storage": "^2.0.0",
"charm": "^1.0.2",
"definitelytyped-header-parser": "3.8.2",
"dtslint": "latest",
"fs-extra": "4.0.0",
"fstream": "^1.0.12",
"longjohn": "^0.2.11",
"moment": "^2.18.1",
"npm": "^6.13.4",
"npm-registry-client": "^8.1.0",
"oboe": "^2.1.3",
"source-map-support": "^0.4.0",
"tar": "^2.2.2",
"tar-stream": "^1.6.2",
"travis-fold": "^0.1.2",
"typescript": "next",
"yargs": "^8.0.2"
},
"devDependencies": {
"@types/charm": "^1.0.0",
"@types/fs-extra": "4.0.0",
"@types/jest": "^23.3.9",
"@types/mz": "^0.0.31",
"@types/node": "^10.11.3",
"@types/oboe": "^2.0.28",
"@types/source-map-support": "^0.4.0",
"@types/tar": "^1.0.27",
"@types/travis-fold": "^0.1.0",
"@types/yargs": "^8.0.1",
"jest": "^24.9.0",
"ts-jest": "^24.2.0",
"tslint": "^5.13.0"
},
"scripts": {
"test": "npm run build && jest && npm run lint",
"build": "node node_modules/typescript/lib/tsc.js",
"watch": "node node_modules/typescript/lib/tsc.js --watch",
"clean": "node -r source-map-support/register bin/clean.js",
"get-definitely-typed": "node -r source-map-support/register bin/get-definitely-typed.js",
"parse": "node -r source-map-support/register bin/parse-definitions.js",
"check": "node -r source-map-support/register bin/check-parse-results.js",
"calculate-versions": "node -r source-map-support/register bin/calculate-versions.js",
"generate": "node -r source-map-support/register bin/generate-packages.js",
"validate": "node -r source-map-support/register bin/validate.js",
"index": "node -r source-map-support/register bin/create-search-index.js",
"publish": "node -r source-map-support/register bin/publish-packages.js",
"publish-dry": "node -r source-map-support/register bin/publish-packages.js --dry",
"publish-registry": "node -r source-map-support/register bin/publish-registry.js",
"upload-blobs": "node -r source-map-support/register bin/upload-blobs.js",
"full": "node -r source-map-support/register bin/full.js",
"full-dry": "node -r source-map-support/register bin/full.js --dry",
"lint": "node node_modules/tslint/bin/tslint --format stylish --project tsconfig.json",
"webhook-dry": "node -r source-map-support/register bin/webhook.js --dry",
"make-server-run": "node -r source-map-support/register bin/make-server-run.js",
"make-production-server-run": "node -r source-map-support/register bin/make-server-run.js --remote",
"test-tsNext": "node -r source-map-support/register bin/tester/test.js --all --tsNext",
"code-owners": "node -r source-map-support/register bin/code-owners.js",
"push-production": "npm run build && git checkout production && git merge master && npm run build && git add bin && git commit -m \"Update bin\" && git push -u origin production"
},
"repository": {
"type": "git",
"url": "git+https://github.com/Microsoft/types-publisher.git"
},
"author": "Microsoft",
"license": "MIT",
"bugs": {
"url": "https://github.com/Microsoft/types-publisher/issues"
},
"engines": {
"node": ">=6.10.0"
},
"homepage": "https://github.com/Microsoft/types-publisher#readme"
}
================================================
FILE: server.js
================================================
require("./bin/webhook.js").default().catch(console.error);
================================================
FILE: src/.vscode/launch.json
================================================
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "attach",
"name": "9229",
"port": 9229,
"sourceMaps": true,
"stopOnEntry": false
}
]
}
================================================
FILE: src/calculate-versions.ts
================================================
import assert = require("assert");
import { FS, getDefinitelyTyped } from "./get-definitely-typed";
import { Options, writeDataFile } from "./lib/common";
import { CachedNpmInfoClient, NpmInfoVersion, UncachedNpmInfoClient, withNpmCache } from "./lib/npm-client";
import { AllPackages, NotNeededPackage, TypingsData } from "./lib/packages";
import { ChangedPackages, ChangedPackagesJson, ChangedTypingJson, Semver, versionsFilename } from "./lib/versions";
import { loggerWithErrors, LoggerWithErrors } from "./util/logging";
import { assertDefined, best, logUncaughtErrors, mapDefined, mapDefinedAsync } from "./util/util";
if (!module.parent) {
const log = loggerWithErrors()[0];
logUncaughtErrors(async () => calculateVersions(await getDefinitelyTyped(Options.defaults, log), new UncachedNpmInfoClient(), log));
}
export default async function calculateVersions(
dt: FS,
uncachedClient: UncachedNpmInfoClient,
log: LoggerWithErrors,
): Promise<ChangedPackages> {
log.info("=== Calculating versions ===");
return withNpmCache(uncachedClient, async client => {
log.info("* Reading packages...");
const packages = await AllPackages.read(dt);
return computeAndSaveChangedPackages(packages, log, client);
});
}
async function computeAndSaveChangedPackages(
allPackages: AllPackages,
log: LoggerWithErrors,
client: CachedNpmInfoClient,
): Promise<ChangedPackages> {
const cp = await computeChangedPackages(allPackages, log, client);
const json: ChangedPackagesJson = {
changedTypings: cp.changedTypings.map(({ pkg: { id }, version, latestVersion }): ChangedTypingJson => ({ id, version, latestVersion })),
changedNotNeededPackages: cp.changedNotNeededPackages.map(p => p.name),
};
await writeDataFile(versionsFilename, json);
return cp;
}
async function computeChangedPackages(
allPackages: AllPackages,
log: LoggerWithErrors,
client: CachedNpmInfoClient,
): Promise<ChangedPackages> {
log.info("# Computing changed packages...");
const changedTypings = await mapDefinedAsync(allPackages.allTypings(), async pkg => {
const { version, needsPublish } = await fetchTypesPackageVersionInfo(pkg, client, /*publish*/ true, log);
if (needsPublish) {
log.info(`Changed: ${pkg.desc}`);
for (const { name } of pkg.packageJsonDependencies) {
assertDefined(
await client.fetchAndCacheNpmInfo(name),
`'${pkg.name}' depends on '${name}' which does not exist on npm. All dependencies must exist.`);
}
const latestVersion = pkg.isLatest ?
undefined :
(await fetchTypesPackageVersionInfo(allPackages.getLatest(pkg), client, /*publish*/ true)).version;
return { pkg, version, latestVersion };
}
return undefined;
});
log.info("# Computing deprecated packages...");
const changedNotNeededPackages = await mapDefinedAsync(allPackages.allNotNeeded(), async pkg => {
if (!await isAlreadyDeprecated(pkg, client, log)) {
assertDefined(
await client.fetchAndCacheNpmInfo(pkg.unescapedName),
`To deprecate '@types/${pkg.name}', '${pkg.unescapedName}' must exist on npm.`);
log.info(`To be deprecated: ${pkg.name}`);
return pkg;
}
return undefined;
});
return { changedTypings, changedNotNeededPackages };
}
async function fetchTypesPackageVersionInfo(
pkg: TypingsData,
client: CachedNpmInfoClient,
canPublish: boolean,
log?: LoggerWithErrors,
): Promise<{ version: string, needsPublish: boolean }> {
let info = client.getNpmInfoFromCache(pkg.fullEscapedNpmName);
let latestVersion = info && getHighestVersionForMajor(info.versions, pkg);
let latestVersionInfo = latestVersion && assertDefined(info!.versions.get(latestVersion.versionString));
if (!latestVersionInfo || latestVersionInfo.typesPublisherContentHash !== pkg.contentHash) {
if (log) { log.info(`Version info not cached for ${pkg.desc}`); }
info = await client.fetchAndCacheNpmInfo(pkg.fullEscapedNpmName);
latestVersion = info && getHighestVersionForMajor(info.versions, pkg);
latestVersionInfo = latestVersion && assertDefined(info!.versions.get(latestVersion.versionString));
if (!latestVersionInfo) { return { version: versionString(pkg, /*patch*/ 0), needsPublish: true }; }
}
if (latestVersionInfo.deprecated) {
// https://github.com/DefinitelyTyped/DefinitelyTyped/pull/22306
assert(
pkg.name === "angular-ui-router" || pkg.name === "ui-router-extras",
`Package ${pkg.name} has been deprecated, so we shouldn't have parsed it. Was it re-added?`);
}
const needsPublish = canPublish && pkg.contentHash !== latestVersionInfo.typesPublisherContentHash;
const patch = needsPublish ? (latestVersion!.minor === pkg.minor ? latestVersion!.patch + 1 : 0) : latestVersion!.patch;
return { version: versionString(pkg, patch), needsPublish };
}
function versionString(pkg: TypingsData, patch: number): string {
return new Semver(pkg.major, pkg.minor, patch).versionString;
}
async function isAlreadyDeprecated(pkg: NotNeededPackage, client: CachedNpmInfoClient, log: LoggerWithErrors): Promise<boolean> {
const cachedInfo = client.getNpmInfoFromCache(pkg.fullEscapedNpmName);
let latestVersion = cachedInfo && assertDefined(cachedInfo.distTags.get("latest"));
let latestVersionInfo = cachedInfo && latestVersion && assertDefined(cachedInfo.versions.get(latestVersion));
if (!latestVersionInfo || !latestVersionInfo.deprecated) {
log.info(`Version info not cached for deprecated package ${pkg.desc}`);
const info = assertDefined(await client.fetchAndCacheNpmInfo(pkg.fullEscapedNpmName));
latestVersion = assertDefined(info.distTags.get("latest"));
latestVersionInfo = assertDefined(info.versions.get(latestVersion));
}
return !!latestVersionInfo.deprecated;
}
function getHighestVersionForMajor(versions: ReadonlyMap<string, NpmInfoVersion>, { major, minor }: TypingsData): Semver | undefined {
const patch = latestPatchMatchingMajorAndMinor(versions.keys(), major, minor);
return patch === undefined ? undefined : new Semver(major, minor, patch);
}
/** Finds the version with matching major/minor with the latest patch version. */
function latestPatchMatchingMajorAndMinor(versions: Iterable<string>, newMajor: number, newMinor: number): number | undefined {
const versionsWithTypings = mapDefined(versions, v => {
const semver = Semver.tryParse(v);
if (!semver) {
return undefined;
}
const { major, minor, patch } = semver;
return major === newMajor && minor === newMinor ? patch : undefined;
});
return best(versionsWithTypings, (a, b) => a > b);
}
export async function getLatestTypingVersion(pkg: TypingsData, client: CachedNpmInfoClient): Promise<string> {
return (await fetchTypesPackageVersionInfo(pkg, client, /*publish*/ false)).version;
}
================================================
FILE: src/check-parse-results.ts
================================================
import { FS, getDefinitelyTyped } from "./get-definitely-typed";
import { Options } from "./lib/common";
import { NpmInfoRawVersions, NpmInfoVersion, UncachedNpmInfoClient } from "./lib/npm-client";
import { AllPackages, formatTypingVersion, TypingsData, TypingVersion } from "./lib/packages";
import { Semver } from "./lib/versions";
import { Logger, logger, loggerWithErrors, writeLog } from "./util/logging";
import { assertDefined, best, logUncaughtErrors, mapDefined, nAtATime } from "./util/util";
if (!module.parent) {
const log = loggerWithErrors()[0];
logUncaughtErrors(
async () => checkParseResults(true, await getDefinitelyTyped(Options.defaults, log), Options.defaults, new UncachedNpmInfoClient()));
}
export default async function checkParseResults(includeNpmChecks: boolean, dt: FS, options: Options, client: UncachedNpmInfoClient): Promise<void> {
const allPackages = await AllPackages.read(dt);
const [log, logResult] = logger();
checkTypeScriptVersions(allPackages);
checkPathMappings(allPackages);
const dependedOn = new Set<string>();
const packages = allPackages.allPackages();
for (const pkg of packages) {
if (pkg instanceof TypingsData) {
for (const dep of pkg.dependencies) {
dependedOn.add(dep.name);
}
for (const dep of pkg.testDependencies) {
dependedOn.add(dep);
}
}
}
if (includeNpmChecks) {
await nAtATime(10, allPackages.allTypings(), pkg => checkNpm(pkg, log, dependedOn, client), {
name: "Checking for typed packages...",
flavor: pkg => pkg.desc,
options,
});
}
await writeLog("conflicts.md", logResult());
}
function checkTypeScriptVersions(allPackages: AllPackages): void {
for (const pkg of allPackages.allTypings()) {
for (const dep of allPackages.allDependencyTypings(pkg)) {
if (dep.minTypeScriptVersion > pkg.minTypeScriptVersion) {
throw new Error(`${pkg.desc} depends on ${dep.desc} but has a lower required TypeScript version.`);
}
}
}
}
function checkPathMappings(allPackages: AllPackages): void {
for (const pkg of allPackages.allTypings()) {
const pathMappings = new Map<string, TypingVersion>(pkg.pathMappings.map(p => [p.packageName, p.version]));
const unusedPathMappings = new Set(pathMappings.keys());
// If A depends on B, and B has path mappings, A must have the same mappings.
for (const dependency of allPackages.allDependencyTypings(pkg)) {
for (const { packageName: transitiveDependencyName, version: transitiveDependencyVersion } of dependency.pathMappings) {
const pathMappingVersion = pathMappings.get(transitiveDependencyName);
if (
pathMappingVersion
&& (
pathMappingVersion.major !== transitiveDependencyVersion.major
|| pathMappingVersion.minor !== transitiveDependencyVersion.minor
)
) {
const expectedPathMapping = `${transitiveDependencyName}/v${formatTypingVersion(transitiveDependencyVersion)}`;
throw new Error(
`${pkg.desc} depends on ${dependency.desc}, which has a path mapping for ${expectedPathMapping}. ` +
`${pkg.desc} must have the same path mappings as its dependencies.`);
}
unusedPathMappings.delete(transitiveDependencyName);
}
unusedPathMappings.delete(dependency.name);
}
for (const unusedPathMapping of unusedPathMappings) {
if (pkg.name !== unusedPathMapping) {
throw new Error(`${pkg.desc} has unused path mapping for ${unusedPathMapping}`);
}
}
}
}
async function checkNpm(
{ major, minor, name, libraryName, projectName, contributors }: TypingsData,
log: Logger,
dependedOn: ReadonlySet<string>,
client: UncachedNpmInfoClient,
): Promise<void> {
if (notNeededExceptions.has(name)) {
return;
}
const info = await client.fetchRawNpmInfo(name); // Gets info for the real package, not the @types package
if (!info) { return; }
const versions = getRegularVersions(info.versions);
const firstTypedVersion = best(mapDefined(versions, ({ hasTypes, version }) => hasTypes ? version : undefined), (a, b) => b.greaterThan(a));
// A package might have added types but removed them later, so check the latest version too
if (firstTypedVersion === undefined || !best(versions, (a, b) => a.version.greaterThan(b.version))!.hasTypes) {
return;
}
const ourVersion = `${major}.${minor}`;
log("");
log(`Typings already defined for ${name} (${libraryName}) as of ${firstTypedVersion.versionString} (our version: ${ourVersion})`);
const contributorUrls = contributors.map(c => {
const gh = "https://github.com/";
return c.url.startsWith(gh) ? `@${c.url.slice(gh.length)}` : `${c.name} (${c.url})`;
}).join(", ");
log(" To fix this:");
log(` git checkout -b not-needed-${name}`);
const yarnargs = [name, firstTypedVersion.versionString, projectName];
if (libraryName !== name) {
yarnargs.push(JSON.stringify(libraryName));
}
log(" yarn not-needed " + yarnargs.join(" "));
log(` git add --all && git commit -m "${name}: Provides its own types" && git push -u origin not-needed-${name}`);
log(` And comment PR: This will deprecate \`@types/${name}\` in favor of just \`${name}\`. CC ${contributorUrls}`);
if (new Semver(major, minor, 0).greaterThan(firstTypedVersion)) {
log(" WARNING: our version is greater!");
}
if (dependedOn.has(name)) {
log(" WARNING: other packages depend on this!");
}
}
export async function packageHasTypes(packageName: string, client: UncachedNpmInfoClient): Promise<boolean> {
const info = assertDefined(await client.fetchRawNpmInfo(packageName));
return versionHasTypes(info.versions[info["dist-tags"].latest]);
}
function getRegularVersions(versions: NpmInfoRawVersions): ReadonlyArray<{ readonly version: Semver; readonly hasTypes: boolean; }> {
return mapDefined(Object.entries(versions), ([versionString, info]) => {
const version = Semver.tryParse(versionString);
return version === undefined ? undefined : { version, hasTypes: versionHasTypes(info) };
});
}
function versionHasTypes(info: NpmInfoVersion): boolean {
return "types" in info || "typings" in info;
}
const notNeededExceptions: ReadonlySet<string> = new Set([
// https://github.com/DefinitelyTyped/DefinitelyTyped/pull/22306
"angular-ui-router", "ui-router-extras",
// Declares to bundle types, but they're also in the `.npmignore` (https://github.com/nkovacic/angular-touchspin/issues/21)
"angular-touchspin",
// "typings" points to the wrong file (https://github.com/Microsoft/Bing-Maps-V8-TypeScript-Definitions/issues/31)
"bingmaps",
// Types are bundled, but not officially released (https://github.com/DefinitelyTyped/DefinitelyTyped/pull/22313#issuecomment-353225893)
"dwt",
// Waiting on some typing errors to be fixed (https://github.com/julien-c/epub/issues/30)
"epub",
// Typings file is not in package.json "files" list (https://github.com/silentmatt/expr-eval/issues/127)
"expr-eval",
// NPM package "express-serve-static-core" isn't a real package -- express-serve-static-core exists only for the purpose of types
"express-serve-static-core",
// Has "typings": "index.d.ts" but does not actually bundle typings. https://github.com/kolodny/immutability-helper/issues/79
"immutability-helper",
// Has `"typings": "compiled/typings/node-mysql-wrapper/node-mysql-wrapper.d.ts",`, but `compiled/typings` doesn't exist.
// Package hasn't updated in 2 years and author seems to have deleted their account, so no chance of being fixed.
"node-mysql-wrapper",
// raspi packages bundle types, but can only be installed on a Raspberry Pi, so they are duplicated to DefinitelyTyped.
// See https://github.com/DefinitelyTyped/DefinitelyTyped/pull/21618
"raspi", "raspi-board", "raspi-gpio", "raspi-i2c", "raspi-led", "raspi-onewire",
"raspi-peripheral", "raspi-pwm", "raspi-serial", "raspi-soft-pwm",
// Declare "typings" but don't actually have them yet (https://github.com/stampit-org/stampit/issues/245)
"stampit",
]);
================================================
FILE: src/clean.ts
================================================
import { removeSync } from "fs-extra";
if (!module.parent) {
clean();
}
export function clean() {
for (const dir of ["data", "logs", "output"]) {
console.log(`Clean ${dir}`);
removeSync(dir);
}
}
================================================
FILE: src/crawl-npm.ts
================================================
import assert = require("assert");
import oboe = require("oboe");
import { packageHasTypes } from "./check-parse-results";
import { Options, writeDataFile } from "./lib/common";
import { UncachedNpmInfoClient } from "./lib/npm-client";
import { npmRegistry } from "./lib/settings";
import ProgressBar, { strProgress } from "./util/progress";
import { filterNAtATimeOrdered, logUncaughtErrors } from "./util/util";
if (!module.parent) {
logUncaughtErrors(main(Options.defaults));
}
/** Prints out every package on NPM with 'types'. */
async function main(options: Options): Promise<void> {
const all = await allNpmPackages();
await writeDataFile("all-npm-packages.json", all);
const client = new UncachedNpmInfoClient();
const allTyped = await filterNAtATimeOrdered(10, all, pkg => packageHasTypes(pkg, client), {
name: "Checking for types...",
flavor: (name, isTyped) => isTyped ? name : undefined,
options,
});
await writeDataFile("all-typed-packages.json", allTyped);
console.log(allTyped.join("\n"));
console.log(`Found ${allTyped.length} typed packages.`);
}
function allNpmPackages(): Promise<string[]> {
const progress = new ProgressBar({ name: "Loading NPM packages..." });
// https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md
const url = `${npmRegistry}-/all`;
const all: string[] = [];
return new Promise<string[]>((resolve, reject) => {
oboe(url)
.node("!.*", (x, path) => {
assert((path as string).length > 0);
if (typeof x !== "number") {
const { name } = x as { name: string };
assert(typeof name === "string" && name.length > 0); // tslint:disable-line strict-type-predicates
progress.update(strProgress(name), name);
all.push(name);
}
return oboe.drop;
})
.done(() => {
progress.done();
resolve(all);
})
.fail(err => { reject(err.thrown); });
});
}
================================================
FILE: src/create-search-index.ts
================================================
import * as yargs from "yargs";
import { getDefinitelyTyped } from "./get-definitely-typed";
import { Options, writeDataFile } from "./lib/common";
import { UncachedNpmInfoClient } from "./lib/npm-client";
import { AllPackages, TypingsData } from "./lib/packages";
import { loggerWithErrors } from "./util/logging";
import { logUncaughtErrors } from "./util/util";
if (!module.parent) {
const log = loggerWithErrors()[0];
const single = yargs.argv.single as string | undefined;
if (single) {
logUncaughtErrors(doSingle(single, new UncachedNpmInfoClient()));
} else {
logUncaughtErrors(
async () => createSearchIndex(await AllPackages.read(await getDefinitelyTyped(Options.defaults, log)), new UncachedNpmInfoClient()));
}
}
export interface SearchRecord {
// types package name
readonly t: string;
// globals
readonly g: ReadonlyArray<string>;
// modules
readonly m: ReadonlyArray<string>;
// project name
readonly p: string;
// library name
readonly l: string;
// downloads in the last month from NPM
readonly d: number;
}
export default async function createSearchIndex(packages: AllPackages, client: UncachedNpmInfoClient): Promise<void> {
console.log("Generating search index...");
const records = await createSearchRecords(packages.allLatestTypings(), client);
console.log("Done generating search index. Writing out data files...");
await writeDataFile("search-index-min.json", records, false);
}
async function doSingle(name: string, client: UncachedNpmInfoClient): Promise<void> {
const pkg = await AllPackages.readSingle(name);
const record = (await createSearchRecords([pkg], client))[0];
console.log(record);
}
async function createSearchRecords(packages: ReadonlyArray<TypingsData>, client: UncachedNpmInfoClient): Promise<ReadonlyArray<SearchRecord>> {
// TODO: Would like to just use pkg.unescapedName unconditionally, but npm doesn't allow scoped packages.
const dl = await client.getDownloads(packages.map((pkg, i) => pkg.name === pkg.unescapedName ? pkg.name : `dummy${i}`));
return packages.map((pkg, i): SearchRecord => ({
p: pkg.projectName,
l: pkg.libraryName,
g: pkg.globals,
t: pkg.name,
m: pkg.declaredModules,
d: dl[i],
})).sort((a, b) => b.d - a.d);
}
================================================
FILE: src/full.ts
================================================
import appInsights = require("applicationinsights");
import * as yargs from "yargs";
import calculateVersions from "./calculate-versions";
import { clean } from "./clean";
import createSearchIndex from "./create-search-index";
import generatePackages from "./generate-packages";
import { getDefinitelyTyped } from "./get-definitely-typed";
import { Options } from "./lib/common";
import { UncachedNpmInfoClient } from "./lib/npm-client";
import parseDefinitions from "./parse-definitions";
import publishPackages from "./publish-packages";
import publishRegistry from "./publish-registry";
import uploadBlobsAndUpdateIssue from "./upload-blobs";
import { Fetcher } from "./util/io";
import { LoggerWithErrors, loggerWithErrors } from "./util/logging";
import { assertDefined, currentTimeStamp, logUncaughtErrors, numberOfOsProcesses } from "./util/util";
import validate from "./validate";
if (!module.parent) {
if (process.env.APPINSIGHTS_INSTRUMENTATIONKEY) {
appInsights.setup();
appInsights.start();
}
const dry = !!yargs.argv.dry;
logUncaughtErrors(full(dry, currentTimeStamp(), process.env.GH_API_TOKEN || "", new Fetcher(), Options.defaults, loggerWithErrors()[0]));
}
export default async function full(
dry: boolean,
timeStamp: string,
githubAccessToken: string,
fetcher: Fetcher,
options: Options,
log: LoggerWithErrors): Promise<void> {
const infoClient = new UncachedNpmInfoClient();
clean();
const dt = await getDefinitelyTyped(options, log);
const allPackages = await parseDefinitions(
dt,
options.parseInParallel
? { nProcesses: numberOfOsProcesses, definitelyTypedPath: assertDefined(options.definitelyTypedPath) }
: undefined,
log);
const changedPackages = await calculateVersions(dt, infoClient, log);
await generatePackages(dt, allPackages, changedPackages);
await createSearchIndex(allPackages, infoClient);
await publishPackages(changedPackages, dry, githubAccessToken, fetcher);
await publishRegistry(dt, allPackages, dry, infoClient);
await validate(dt);
if (!dry) {
await uploadBlobsAndUpdateIssue(timeStamp);
}
}
================================================
FILE: src/generate-packages.test.ts
================================================
import { createNotNeededPackageJSON, createPackageJSON, createReadme, getLicenseFileText } from "./generate-packages";
import { Registry } from "./lib/common";
import { AllPackages, License, NotNeededPackage, readNotNeededPackages, TypesDataFile, TypingsData, TypingsDataRaw } from "./lib/packages";
import { createMockDT } from "./mocks";
import { testo } from "./util/test";
function createRawPackage(license: License): TypingsDataRaw {
return {
libraryName: "jquery",
typingsPackageName: "jquery",
dependencies: [{ name: "madeira", version: { major: 1 } }],
testDependencies: [],
pathMappings: [],
contributors: [{ name: "A", url: "b@c.d", githubUsername: "e" }],
libraryMajorVersion: 1,
libraryMinorVersion: 0,
minTsVersion: "3.0",
typesVersions: [],
files: ["index.d.ts", "jquery.test.ts"],
license,
packageJsonDependencies: [{ name: "balzac", version: "~3" }],
contentHash: "11",
projectName: "jquery.org",
globals: [],
declaredModules: ["jquery"],
};
}
function createTypesData(): TypesDataFile {
return {
jquery: {
1: createRawPackage(License.MIT),
},
madeira: {
1: createRawPackage(License.Apache20),
},
};
}
function createUnneededPackage() {
return new NotNeededPackage({
libraryName: "absalom",
typingsPackageName: "absalom",
asOfVersion: "1.1.1",
sourceRepoURL: "https://github.com/aardwulf/absalom",
});
}
testo({
mitLicenseText() {
const typing = new TypingsData(createRawPackage(License.MIT), /*isLatest*/ true);
expect(getLicenseFileText(typing)).toEqual(expect.stringContaining("MIT License"));
},
apacheLicenseText() {
const typing = new TypingsData(createRawPackage(License.Apache20), /*isLatest*/ true);
expect(getLicenseFileText(typing)).toEqual(expect.stringContaining("Apache License, Version 2.0"));
},
basicReadme() {
const typing = new TypingsData(createRawPackage(License.Apache20), /*isLatest*/ true);
expect(createReadme(typing)).toEqual(expect.stringContaining("This package contains type definitions for"));
},
readmeContainsProjectName() {
const typing = new TypingsData(createRawPackage(License.Apache20), /*isLatest*/ true);
expect(createReadme(typing)).toEqual(expect.stringContaining("jquery.org"));
},
readmeOneDependency() {
const typing = new TypingsData(createRawPackage(License.Apache20), /*isLatest*/ true);
expect(createReadme(typing)).toEqual(expect.stringContaining("Dependencies: [@types/madeira](https://npmjs.com/package/@types/madeira)"));
},
readmeNoGlobals() {
const typing = new TypingsData(createRawPackage(License.Apache20), /*isLatest*/ true);
expect(createReadme(typing)).toEqual(expect.stringContaining("Global values: none"));
},
basicPackageJson() {
const packages = AllPackages.from(createTypesData(), readNotNeededPackages(createMockDT().fs));
const typing = new TypingsData(createRawPackage(License.MIT), /*isLatest*/ true);
expect(createPackageJSON(typing, "1.0", packages, Registry.NPM)).toEqual(`{
"name": "@types/jquery",
"version": "1.0",
"description": "TypeScript definitions for jquery",
"license": "MIT",
"contributors": [
{
"name": "A",
"url": "b@c.d",
"githubUsername": "e"
}
],
"main": "",
"types": "index.d.ts",
"repository": {
"type": "git",
"url": "https://github.com/DefinitelyTyped/DefinitelyTyped.git",
"directory": "types/jquery"
},
"scripts": {},
"dependencies": {
"@types/madeira": "^1",
"balzac": "~3"
},
"typesPublisherContentHash": "11",
"typeScriptVersion": "3.0"
}`);
},
githubPackageJsonName() {
const packages = AllPackages.from(createTypesData(), readNotNeededPackages(createMockDT().fs));
const typing = new TypingsData(createRawPackage(License.MIT), /*isLatest*/ true);
expect(createPackageJSON(typing, "1.0", packages, Registry.Github)).toEqual(
expect.stringContaining('"name": "@types/jquery"'));
},
githubPackageJsonRegistry() {
const packages = AllPackages.from(createTypesData(), readNotNeededPackages(createMockDT().fs));
const typing = new TypingsData(createRawPackage(License.MIT), /*isLatest*/ true);
const s = createPackageJSON(typing, "1.0", packages, Registry.Github);
expect(s).toEqual(expect.stringContaining("publishConfig"));
expect(s).toEqual(expect.stringContaining('"registry": "https://npm.pkg.github.com/"'));
},
basicNotNeededPackageJson() {
const s = createNotNeededPackageJSON(createUnneededPackage(), Registry.NPM);
expect(s).toEqual(`{
"name": "@types/absalom",
"version": "1.1.1",
"typings": null,
"description": "Stub TypeScript definitions entry for absalom, which provides its own types definitions",
"main": "",
"scripts": {},
"author": "",
"repository": "https://github.com/aardwulf/absalom",
"license": "MIT",
"dependencies": {
"absalom": "*"
}
}`);
},
scopedNotNeededPackageJson() {
const scopedUnneeded = new NotNeededPackage({
libraryName: "@google-cloud/pubsub",
typingsPackageName: "google-cloud__pubsub",
asOfVersion: "0.26.0",
sourceRepoURL: "https://github.com/googleapis/nodejs-storage",
});
const s = createNotNeededPackageJSON(scopedUnneeded, Registry.NPM);
expect(s).toEqual(`{
"name": "@types/google-cloud__pubsub",
"version": "0.26.0",
"typings": null,
"description": "Stub TypeScript definitions entry for @google-cloud/pubsub, which provides its own types definitions",
"main": "",
"scripts": {},
"author": "",
"repository": "https://github.com/googleapis/nodejs-storage",
"license": "MIT",
"dependencies": {
"@google-cloud/pubsub": "*"
}
}`);
},
githubNotNeededPackageJson() {
const s = createNotNeededPackageJSON(createUnneededPackage(), Registry.Github);
expect(s).toEqual(expect.stringContaining("@types"));
expect(s).toEqual(expect.stringContaining("npm.pkg.github.com"));
},
});
================================================
FILE: src/generate-packages.ts
================================================
import { makeTypesVersionsForPackageJson } from "definitelytyped-header-parser";
import { emptyDir, mkdir, mkdirp, readFileSync } from "fs-extra";
import * as path from "path";
import * as yargs from "yargs";
import { FS, getDefinitelyTyped } from "./get-definitely-typed";
import { Options, Registry } from "./lib/common";
import { CachedNpmInfoClient, UncachedNpmInfoClient, withNpmCache } from "./lib/npm-client";
import {
AllPackages, AnyPackage, DependencyVersion, formatTypingVersion, getFullNpmName, License, NotNeededPackage, PackageJsonDependency, TypingsData,
} from "./lib/packages";
import { outputDirPath, sourceBranch } from "./lib/settings";
import { ChangedPackages, readChangedPackages, skipBadPublishes } from "./lib/versions";
import { writeFile } from "./util/io";
import { logger, Logger, loggerWithErrors, writeLog } from "./util/logging";
import { writeTgz } from "./util/tgz";
import { assertNever, joinPaths, logUncaughtErrors, sortObjectKeys } from "./util/util";
const mitLicense = readFileSync(joinPaths(__dirname, "..", "LICENSE"), "utf-8");
if (!module.parent) {
const tgz = !!yargs.argv.tgz;
logUncaughtErrors(async () => {
const log = loggerWithErrors()[0];
const dt = await getDefinitelyTyped(Options.defaults, log);
const allPackages = await AllPackages.read(dt);
await generatePackages(dt, allPackages, await readChangedPackages(allPackages), tgz);
});
}
export default async function generatePackages(dt: FS, allPackages: AllPackages, changedPackages: ChangedPackages, tgz = false): Promise<void> {
const [log, logResult] = logger();
log("\n## Generating packages");
await emptyDir(outputDirPath);
for (const { pkg, version } of changedPackages.changedTypings) {
await generateTypingPackage(pkg, allPackages, version, dt);
if (tgz) {
await writeTgz(pkg.outputDirectory, `${pkg.outputDirectory}.tgz`);
}
log(` * ${pkg.desc}`);
}
log("## Generating deprecated packages");
await withNpmCache(new UncachedNpmInfoClient(), async client => {
for (const pkg of changedPackages.changedNotNeededPackages) {
log(` * ${pkg.libraryName}`);
await generateNotNeededPackage(pkg, client, log);
}
});
await writeLog("package-generator.md", logResult());
}
async function generateTypingPackage(typing: TypingsData, packages: AllPackages, version: string, dt: FS): Promise<void> {
const typesDirectory = dt.subDir("types").subDir(typing.name);
const packageFS = typing.isLatest ? typesDirectory : typesDirectory.subDir(typing.versionDirectoryName!);
await writeCommonOutputs(typing, createPackageJSON(typing, version, packages, Registry.NPM), createReadme(typing), Registry.NPM);
await writeCommonOutputs(typing, createPackageJSON(typing, version, packages, Registry.Github), createReadme(typing), Registry.Github);
await Promise.all(
typing.files.map(async file => writeFile(await outputFilePath(typing, Registry.NPM, file), packageFS.readFile(file))));
await Promise.all(
typing.files.map(async file => writeFile(await outputFilePath(typing, Registry.Github, file), packageFS.readFile(file))));
}
async function generateNotNeededPackage(pkg: NotNeededPackage, client: CachedNpmInfoClient, log: Logger): Promise<void> {
pkg = skipBadPublishes(pkg, client, log);
await writeCommonOutputs(pkg, createNotNeededPackageJSON(pkg, Registry.NPM), pkg.readme(), Registry.NPM);
await writeCommonOutputs(pkg, createNotNeededPackageJSON(pkg, Registry.Github), pkg.readme(), Registry.Github);
}
async function writeCommonOutputs(pkg: AnyPackage, packageJson: string, readme: string, registry: Registry): Promise<void> {
await mkdir(pkg.outputDirectory + (registry === Registry.Github ? "-github" : ""));
await Promise.all([
writeOutputFile("package.json", packageJson),
writeOutputFile("README.md", readme),
writeOutputFile("LICENSE", getLicenseFileText(pkg)),
]);
async function writeOutputFile(filename: string, content: string): Promise<void> {
await writeFile(await outputFilePath(pkg, registry, filename), content);
}
}
async function outputFilePath(pkg: AnyPackage, registry: Registry, filename: string): Promise<string> {
const full = joinPaths(pkg.outputDirectory + (registry === Registry.Github ? "-github" : ""), filename);
const dir = path.dirname(full);
if (dir !== pkg.outputDirectory) {
await mkdirp(dir);
}
return full;
}
interface Dependencies { [name: string]: string; }
export function createPackageJSON(typing: TypingsData, version: string, packages: AllPackages, registry: Registry): string {
// Use the ordering of fields from https://docs.npmjs.com/files/package.json
const out: {} = {
name: typing.fullNpmName,
version,
description: `TypeScript definitions for ${typing.libraryName}`,
// keywords,
// homepage,
// bugs,
license: typing.license,
contributors: typing.contributors,
main: "",
types: "index.d.ts",
typesVersions: makeTypesVersionsForPackageJson(typing.typesVersions),
repository: {
type: "git",
url: registry === Registry.Github
? "https://github.com/types/_definitelytypedmirror.git"
: "https://github.com/DefinitelyTyped/DefinitelyTyped.git",
directory: `types/${typing.name}`,
},
scripts: {},
dependencies: getDependencies(typing.packageJsonDependencies, typing, packages),
typesPublisherContentHash: typing.contentHash,
typeScriptVersion: typing.minTypeScriptVersion,
};
if (registry === Registry.Github) {
(out as any).publishConfig = { registry: "https://npm.pkg.github.com/" };
}
return JSON.stringify(out, undefined, 4);
}
const definitelyTypedURL = "https://github.com/DefinitelyTyped/DefinitelyTyped";
/** Adds inferred dependencies to `dependencies`, if they are not already specified in either `dependencies` or `peerDependencies`. */
function getDependencies(packageJsonDependencies: ReadonlyArray<PackageJsonDependency>, typing: TypingsData, allPackages: AllPackages): Dependencies {
const dependencies: Dependencies = {};
for (const { name, version } of packageJsonDependencies) {
dependencies[name] = version;
}
for (const dependency of typing.dependencies) {
const typesDependency = getFullNpmName(dependency.name);
// A dependency "foo" is already handled if we already have a dependency on the package "foo" or "@types/foo".
if (!packageJsonDependencies.some(d => d.name === dependency.name || d.name === typesDependency) && allPackages.hasTypingFor(dependency)) {
dependencies[typesDependency] = dependencySemver(dependency.version);
}
}
return sortObjectKeys(dependencies);
}
function dependencySemver(dependency: DependencyVersion): string {
return dependency === "*" ? dependency : "^" + formatTypingVersion(dependency);
}
export function createNotNeededPackageJSON(
{
libraryName,
license,
unescapedName,
fullNpmName,
sourceRepoURL,
version,
}: NotNeededPackage,
registry: Registry,
): string {
const out = {
name: fullNpmName,
version: version.versionString,
typings: null, // tslint:disable-line no-null-keyword
description: `Stub TypeScript definitions entry for ${libraryName}, which provides its own types definitions`,
main: "",
scripts: {},
author: "",
repository: registry === Registry.NPM ? sourceRepoURL : "https://github.com/types/_definitelytypedmirror.git",
license,
// No `typings`, that's provided by the dependency.
dependencies: {
[unescapedName]: "*",
},
};
if (registry === Registry.Github) {
(out as any).publishConfig = { registry: "https://npm.pkg.github.com/" };
}
return JSON.stringify(out, undefined, 4);
}
export function createReadme(typing: TypingsData): string {
const lines: string[] = [];
lines.push("# Installation");
lines.push(`> \`npm install --save ${typing.fullNpmName}\``);
lines.push("");
lines.push("# Summary");
if (typing.projectName) {
lines.push(`This package contains type definitions for ${typing.libraryName} (${typing.projectName}).`);
} else {
lines.push(`This package contains type definitions for ${typing.libraryName}.`);
}
lines.push("");
lines.push("# Details");
lines.push(`Files were exported from ${definitelyTypedURL}/tree/${sourceBranch}/types/${typing.subDirectoryPath}.`);
lines.push("");
lines.push("### Additional Details");
lines.push(` * Last updated: ${(new Date()).toUTCString()}`);
const dependencies = Array.from(typing.dependencies).map(d => getFullNpmName(d.name));
lines.push(` * Dependencies: ${dependencies.length ? dependencies.map(d => `[${d}](https://npmjs.com/package/${d})`).join(", ") : "none"}`);
lines.push(` * Global values: ${typing.globals.length ? typing.globals.map(g => `\`${g}\``).join(", ") : "none"}`);
lines.push("");
lines.push("# Credits");
const contributors = typing.contributors.map(({ name, url }) => `[${name}](${url})`).join(", ").replace(/, ([^,]+)$/, ", and $1");
lines.push(`These definitions were written by ${contributors}.`);
lines.push("");
return lines.join("\r\n");
}
export function getLicenseFileText(typing: AnyPackage): string {
switch (typing.license) {
case License.MIT:
return mitLicense;
case License.Apache20:
return apacheLicense(typing);
default:
throw assertNever(typing);
}
}
function apacheLicense(typing: TypingsData): string {
const year = new Date().getFullYear();
const names = typing.contributors.map(c => c.name);
// tslint:disable max-line-length
return `Copyright ${year} ${names.join(", ")}
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.`;
// tslint:enable max-line-length
}
================================================
FILE: src/get-definitely-typed.test.ts
================================================
import { Dir, FS, getDefinitelyTyped, InMemoryDT } from "./get-definitely-typed";
import { Options } from "./lib/common";
import { loggerWithErrors } from "./util/logging";
import { testo } from "./util/test";
testo({
async downloadDefinitelyTyped() {
const dt = await getDefinitelyTyped(Options.azure, loggerWithErrors()[0]);
expect(dt.exists("types")).toBe(true);
expect(dt.exists("buncho")).toBe(false);
},
createDirs() {
const root = new Dir(undefined);
root.set("file1.txt", "ok");
expect(root.has("file1.txt")).toBe(true);
expect(root.get("file1.txt")).toBe("ok");
},
simpleMemoryFS() {
const root = new Dir(undefined);
root.set("file1.txt", "ok");
const dir = root.subdir("sub1");
dir.set("file2.txt", "x");
const fs: FS = new InMemoryDT(root, "test/");
expect(fs.exists("file1.txt")).toBe(true);
expect(fs.readFile("file1.txt")).toBe("ok");
expect(fs.readFile("sub1/file2.txt")).toBe("x");
},
});
================================================
FILE: src/get-definitely-typed.ts
================================================
import appInsights = require("applicationinsights");
import assert = require("assert");
import { ensureDir, pathExistsSync, readdirSync, statSync } from "fs-extra";
import https = require("https");
import tarStream = require("tar-stream");
import * as yargs from "yargs";
import * as zlib from "zlib";
import { Options } from "./lib/common";
import { dataDirPath, definitelyTypedZipUrl } from "./lib/settings";
import { readFileSync, readJsonSync, stringOfStream } from "./util/io";
import { LoggerWithErrors, loggerWithErrors } from "./util/logging";
import { assertDefined, exec, joinPaths, logUncaughtErrors, withoutStart } from "./util/util";
/**
* Readonly filesystem.
* Paths provided to these methods should be relative to the FS object's root but not start with '/' or './'.
*/
export interface FS {
/**
* Alphabetically sorted list of files and subdirectories.
* If dirPath is missing, reads the root.
*/
readdir(dirPath?: string): ReadonlyArray<string>;
readJson(path: string): unknown;
readFile(path: string): string;
isDirectory(dirPath: string): boolean;
exists(path: string): boolean;
/** FileSystem rooted at a child directory. */
subDir(path: string): FS;
/** Representation of current location, for debugging. */
debugPath(): string;
}
if (!module.parent) {
if (process.env.APPINSIGHTS_INSTRUMENTATIONKEY) {
appInsights.setup();
appInsights.start();
}
const dry = !!yargs.argv.dry;
console.log("gettingDefinitelyTyped: " + (dry ? "from github" : "locally"));
logUncaughtErrors(async () => {
const dt = await getDefinitelyTyped(dry ? Options.azure : Options.defaults, loggerWithErrors()[0]);
assert(dt.exists("types"));
assert(!(dt.exists("buncho")));
});
}
export async function getDefinitelyTyped(options: Options, log: LoggerWithErrors): Promise<FS> {
if (options.definitelyTypedPath === undefined) {
log.info("Downloading Definitely Typed ...");
await ensureDir(dataDirPath);
return downloadAndExtractFile(definitelyTypedZipUrl);
}
const { error, stderr, stdout } = await exec("git diff --name-only", options.definitelyTypedPath);
if (error) { throw error; }
if (stderr) { throw new Error(stderr); }
if (stdout) { throw new Error(`'git diff' should be empty. Following files changed:\n${stdout}`); }
log.info(`Using local Definitely Typed at ${options.definitelyTypedPath}.`);
return new DiskFS(`${options.definitelyTypedPath}/`);
}
export function getLocallyInstalledDefinitelyTyped(path: string): FS {
return new DiskFS(`${path}/`);
}
function downloadAndExtractFile(url: string): Promise<FS> {
return new Promise<FS>((resolve, reject) => {
const root = new Dir(undefined);
function insertFile(path: string, content: string): void {
const components = path.split("/");
const baseName = assertDefined(components.pop());
let dir = root;
for (const component of components) {
dir = dir.subdir(component);
}
dir.set(baseName, content);
}
https.get(url, response => {
const extract = tarStream.extract();
response.pipe(zlib.createGunzip()).pipe(extract);
interface Header {
readonly name: string;
readonly type: "file" | "directory";
}
extract.on("entry", (header: Header, stream: NodeJS.ReadableStream, next: () => void) => {
const name = assertDefined(withoutStart(header.name, "DefinitelyTyped-master/"));
switch (header.type) {
case "file":
stringOfStream(stream, name).then(s => {
insertFile(name, s);
next();
}).catch(reject);
break;
case "directory":
next();
break;
default:
throw new Error(`Unexpected file system entry kind ${header.type}`);
}
});
extract.on("error", reject);
extract.on("finish", () => { resolve(new InMemoryDT(root.finish(), "")); });
});
});
}
interface ReadonlyDir extends ReadonlyMap<string, ReadonlyDir | string> {
readonly parent: Dir | undefined;
}
// Map entries are Dir for directory and string for file.
export class Dir extends Map<string, Dir | string> implements ReadonlyDir {
constructor(readonly parent: Dir | undefined) { super(); }
subdir(name: string): Dir {
const x = this.get(name);
if (x !== undefined) {
if (typeof x === "string") {
throw new Error(`File ${name} has same name as a directory?`);
}
return x;
}
const res = new Dir(this);
this.set(name, res);
return res;
}
finish(): Dir {
const out = new Dir(this.parent);
for (const key of Array.from(this.keys()).sort()) {
const subDirOrFile = this.get(key)!;
out.set(key, typeof subDirOrFile === "string" ? subDirOrFile : subDirOrFile.finish());
}
return out;
}
}
export class InMemoryDT implements FS {
/** pathToRoot is just for debugging */
constructor(readonly curDir: ReadonlyDir, readonly pathToRoot: string) {}
private tryGetEntry(path: string): ReadonlyDir | string | undefined {
validatePath(path);
if (path === "") {
return this.curDir;
}
const components = path.split("/");
const baseName = assertDefined(components.pop());
let dir = this.curDir;
for (const component of components) {
const entry = component === ".." ? dir.parent : dir.get(component);
if (entry === undefined) {
return undefined;
}
if (!(entry instanceof Dir)) {
throw new Error(`No file system entry at ${this.pathToRoot}/${path}. Siblings are: ${Array.from(dir.keys()).toString()}`);
}
dir = entry;
}
return dir.get(baseName);
}
private getEntry(path: string): ReadonlyDir | string {
const entry = this.tryGetEntry(path);
if (entry === undefined) { throw new Error(`No file system entry at ${this.pathToRoot}/${path}`); }
return entry;
}
private getDir(dirPath: string): Dir {
const res = this.getEntry(dirPath);
if (!(res instanceof Dir)) {
throw new Error(`${this.pathToRoot}/${dirPath} is a file, not a directory.`);
}
return res;
}
readFile(filePath: string): string {
const res = this.getEntry(filePath);
if (typeof res !== "string") {
throw new Error(`${this.pathToRoot}/${filePath} is a directory, not a file.`);
}
return res;
}
readdir(dirPath?: string): ReadonlyArray<string> {
return Array.from((dirPath === undefined ? this.curDir : this.getDir(dirPath)).keys());
}
readJson(path: string): unknown {
return JSON.parse(this.readFile(path)) as unknown;
}
isDirectory(path: string): boolean {
return typeof this.getEntry(path) !== "string";
}
exists(path: string): boolean {
return this.tryGetEntry(path) !== undefined;
}
subDir(path: string): FS {
return new InMemoryDT(this.getDir(path), joinPaths(this.pathToRoot, path));
}
debugPath(): string {
return this.pathToRoot;
}
}
class DiskFS implements FS {
constructor(private readonly rootPrefix: string) {
assert(rootPrefix.endsWith("/"));
}
private getPath(path: string | undefined): string {
if (path === undefined) {
return this.rootPrefix;
}
validatePath(path);
return this.rootPrefix + path;
}
readdir(dirPath?: string): ReadonlyArray<string> {
return readdirSync(this.getPath(dirPath)).sort().filter(name => name !== ".DS_Store");
}
isDirectory(dirPath: string): boolean {
return statSync(this.getPath(dirPath)).isDirectory();
}
readJson(path: string): unknown {
return readJsonSync(this.getPath(path));
}
readFile(path: string): string {
return readFileSync(this.getPath(path));
}
exists(path: string): boolean {
return pathExistsSync(this.getPath(path));
}
subDir(path: string): FS {
return new DiskFS(`${this.rootPrefix}${path}/`);
}
debugPath(): string {
return this.rootPrefix.slice(0, this.rootPrefix.length - 1); // remove trailing '/'
}
}
/** FS only handles simple paths like `foo/bar` or `../foo`. No `./foo` or `/foo`. */
function validatePath(path: string): void {
if (path.startsWith(".") && path !== ".editorconfig" && !path.startsWith("../")) {
throw new Error(`${path}: filesystem doesn't support paths of the form './x'.`);
}
if (path.startsWith("/")) {
throw new Error(`${path}: filesystem doesn't support paths of the form '/xxx'.`);
}
if (path.endsWith("/")) {
throw new Error(`${path}: filesystem doesn't support paths of the form 'xxx/'.`);
}
}
================================================
FILE: src/index.ts
================================================
export { getDefinitelyTyped } from "./get-definitely-typed";
export { withNpmCache, CachedNpmInfoClient, NpmPublishClient, UncachedNpmInfoClient } from "./lib/npm-client";
export { AllPackages } from "./lib/packages";
export { clean } from "./clean";
export { getLatestTypingVersion } from "./calculate-versions";
export { default as parseDefinitions } from "./parse-definitions";
export { parseNProcesses } from "./tester/test-runner";
export { consoleLogger, loggerWithErrors } from "./util/logging";
export { logUncaughtErrors, nAtATime } from "./util/util";
export { updateLatestTag, updateTypeScriptVersionTags } from "./lib/package-publisher";
================================================
FILE: src/lib/azure-container.ts
================================================
import { BlobService, common, createBlobService, ErrorOrResponse, ErrorOrResult } from "azure-storage";
import * as fs from "fs";
import * as https from "https";
import { streamDone, streamOfString, stringOfStream } from "../util/io";
import { gzip, unGzip } from "../util/tgz";
import { parseJson } from "../util/util";
import { getSecret, Secret } from "./secrets";
import { azureContainer, azureStorageAccount } from "./settings";
export default class BlobWriter {
static async create(): Promise<BlobWriter> {
return new BlobWriter(createBlobService(azureStorageAccount, await getSecret(Secret.AZURE_STORAGE_ACCESS_KEY)));
}
private constructor(private readonly service: BlobService) {}
setCorsProperties(): Promise<void> {
const properties: common.models.ServicePropertiesResult.ServiceProperties = {
Cors: {
CorsRule: [
{
AllowedOrigins: ["*"],
AllowedMethods: ["GET"],
AllowedHeaders: [],
ExposedHeaders: [],
MaxAgeInSeconds: 60 * 60 * 24, // 1 day
},
],
},
};
return promisifyErrorOrResponse(cb => { this.service.setServiceProperties(properties, cb); });
}
ensureCreated(options: BlobService.CreateContainerOptions): Promise<void> {
return promisifyErrorOrResult<BlobService.ContainerResult>(cb => {
this.service.createContainerIfNotExists(azureContainer, options, cb);
}) as unknown as Promise<void>;
}
createBlobFromFile(blobName: string, fileName: string): Promise<void> {
return this.createBlobFromStream(blobName, fs.createReadStream(fileName));
}
createBlobFromText(blobName: string, text: string): Promise<void> {
return this.createBlobFromStream(blobName, streamOfString(text));
}
async listBlobs(prefix: string): Promise<BlobService.BlobResult[]> {
const once = (tkn: common.ContinuationToken | undefined) =>
promisifyErrorOrResult<BlobService.ListBlobsResult>(cb => {
this.service.listBlobsSegmentedWithPrefix(azureContainer, prefix, tkn!, cb);
});
const out: BlobService.BlobResult[] = [];
let token: common.ContinuationToken | undefined;
do {
const {entries, continuationToken}: BlobService.ListBlobsResult = await once(token);
out.push(...entries);
token = continuationToken;
} while (token);
return out;
}
deleteBlob(blobName: string): Promise<void> {
return promisifyErrorOrResponse(cb => {
this.service.deleteBlob(azureContainer, blobName, cb);
});
}
private createBlobFromStream(blobName: string, stream: NodeJS.ReadableStream): Promise<void> {
const options: BlobService.CreateBlobRequestOptions = {
contentSettings: {
contentEncoding: "gzip",
contentType: "application/json; charset=utf-8",
},
};
// Remove `undefined!` once https://github.com/Azure/azure-storage-node/pull/267 is in
return streamDone(gzip(stream).pipe(this.service.createWriteStreamToBlockBlob(azureContainer, blobName, options, undefined!)));
}
}
export async function readBlob(blobName: string): Promise<string> {
return new Promise<string>((resolve, reject) => {
const url = urlOfBlob(blobName);
const req = https.get(url, res => {
switch (res.statusCode) {
case 200:
if (res.headers["content-encoding"] !== "gzip") {
reject(new Error(`${url} is not gzipped`));
} else {
resolve(stringOfStream(unGzip(res), blobName));
}
break;
default:
reject(new Error(`Can't get ${url}: ${res.statusCode} error`));
}
});
req.on("error", reject);
});
}
export async function readJsonBlob(blobName: string): Promise<object> {
return parseJson(await readBlob(blobName));
}
export function urlOfBlob(blobName: string): string {
return `https://${azureContainer}.blob.core.windows.net/${azureContainer}/${blobName}`;
}
function promisifyErrorOrResult<A>(callsBack: (x: ErrorOrResult<A>) => void): Promise<A> {
return new Promise<A>((resolve, reject) => {
callsBack((err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
});
});
}
function promisifyErrorOrResponse(callsBack: (x: ErrorOrResponse) => void): Promise<void> {
return new Promise<void>((resolve, reject) => {
callsBack(err => {
if (err) {
reject(err);
} else {
resolve();
}
});
});
}
================================================
FILE: src/lib/blob-uploader.ts
================================================
import assert = require("assert");
import { readdir } from "fs-extra";
import * as path from "path";
import { Logger, logger, logPath, writeLog } from "../util/logging";
import { joinPaths, unique } from "../util/util";
import BlobWriter, { urlOfBlob } from "./azure-container";
const maxNumberOfOldLogsDirectories = 5;
export default async function uploadBlobsAndUpdateIssue(timeStamp: string): Promise<void> {
const container = await BlobWriter.create();
await container.ensureCreated({ publicAccessLevel: "blob" });
await container.setCorsProperties();
const [dataUrls, logUrls] = await uploadBlobs(container, timeStamp);
await uploadIndex(container, timeStamp, dataUrls, logUrls);
}
// View uploaded files at: https://ms.portal.azure.com under "typespublisher"
async function uploadBlobs(container: BlobWriter, timeStamp: string): Promise<[string[], string[]]> {
const [log, logResult] = logger();
const [dataUrls, logUrls] = await Promise.all([
await uploadDirectory(container, "data", "data", log),
await uploadLogs(container, timeStamp, log),
]);
// Finally, output blob logs and upload them.
const blobLogs = "upload-blobs.md";
await writeLog(blobLogs, logResult());
logUrls.push(await uploadFile(container, `${logsUploadedLocation(timeStamp)}/${blobLogs}`, logPath(blobLogs)));
return [dataUrls, logUrls];
}
const logsDirectoryName = "logs";
const logsPrefix = `${logsDirectoryName}/`;
function logsUploadedLocation(timeStamp: string): string {
return logsPrefix + timeStamp;
}
async function uploadLogs(container: BlobWriter, timeStamp: string, log: Logger): Promise<string[]> {
await removeOldDirectories(container, logsPrefix, maxNumberOfOldLogsDirectories - 1, log);
return uploadDirectory(container, logsUploadedLocation(timeStamp), logsDirectoryName, log, f => f !== "upload-blobs.md");
}
async function uploadDirectory(
container: BlobWriter, uploadedDirPath: string, dirPath: string, log: Logger,
filter?: (fileName: string) => boolean): Promise<string[]> {
let files = await readdir(dirPath);
if (filter) {
files = files.filter(filter);
}
return Promise.all(files.map(fileName => {
const fullPath = joinPaths(dirPath, fileName);
const blobName = joinPaths(uploadedDirPath, fileName);
return logAndUploadFile(container, blobName, fullPath, log);
}));
}
async function logAndUploadFile(container: BlobWriter, blobName: string, filePath: string, log: Logger): Promise<string> {
const url = urlOfBlob(blobName);
log(`Uploading ${filePath} to ${url}`);
await container.createBlobFromFile(blobName, filePath);
return url;
}
async function uploadFile(container: BlobWriter, blobName: string, filePath: string): Promise<string> {
const url = urlOfBlob(blobName);
await container.createBlobFromFile(blobName, filePath);
return url;
}
async function deleteDirectory(container: BlobWriter, uploadedDirPath: string, log: Logger): Promise<void> {
const blobs = await container.listBlobs(uploadedDirPath);
const blobNames = blobs.map(b => b.name);
log(`Deleting directory ${uploadedDirPath}: delete files ${blobNames.toString()}`);
await Promise.all(blobNames.map(b => container.deleteBlob(b)));
}
async function removeOldDirectories(container: BlobWriter, prefix: string, maxDirectories: number, log: Logger): Promise<void> {
const list = await container.listBlobs(prefix);
const dirNames = unique(list.map(({name}) => {
assert(name.startsWith(prefix));
return path.dirname(name.slice(prefix.length));
}));
if (dirNames.length <= maxDirectories) {
log(`No need to remove old directories: have ${dirNames.length}, can go up to ${maxDirectories}.`);
return;
}
// For ISO 8601 times, sorting lexicographically *is* sorting by time.
const sortedNames = dirNames.sort();
const toDelete = sortedNames.slice(0, sortedNames.length - maxDirectories);
log(`Too many old logs, so removing the following directories: [${toDelete.toString()}]`);
await Promise.all(toDelete.map(d => deleteDirectory(container, prefix + d, log)));
}
// Provides links to the latest blobs.
// These are at: https://typespublisher.blob.core.windows.net/typespublisher/index.html
function uploadIndex(container: BlobWriter, timeStamp: string, dataUrls: ReadonlyArray<string>, logUrls: ReadonlyArray<string>): Promise<void> {
return container.createBlobFromText("index.html", createIndex());
function createIndex(): string {
const lines: string[] = [];
lines.push("<html><head></head><body>");
lines.push(`<h3>Here is the latest data as of **${timeStamp}**:</h3>`);
lines.push("<h4>Data</h4>");
lines.push(...dataUrls.map(link));
lines.push("<h4>Logs</h4>");
lines.push(...logUrls.map(link));
lines.push("</body></html>");
return lines.join("\n");
function link(url: string): string {
const short = url.slice(url.lastIndexOf("/") + 1);
return `<li><a href='${url}'>${short}</a></li>`;
}
}
}
================================================
FILE: src/lib/common.ts
================================================
import { ensureDir } from "fs-extra";
import { readJson, writeJson } from "../util/io";
import { joinPaths } from "../util/util";
import { dataDirPath } from "./settings";
if (process.env.LONGJOHN) {
console.log("=== USING LONGJOHN ===");
const longjohn = require("longjohn") as { async_trace_limit: number }; // tslint:disable-line no-var-requires
longjohn.async_trace_limit = -1; // unlimited
}
/** Which registry to publish to */
export enum Registry {
/** types-registry and @types/* on NPM */
NPM,
/** @definitelytyped/types-registry and @types/* on Github */
Github,
}
/** Settings that may be determined dynamically. */
export interface Options {
/**
* e.g. '../DefinitelyTyped'
* This is overridden to `cwd` when running the tester, as that is run from within DefinitelyTyped.
* If undefined, downloads instead.
*/
readonly definitelyTypedPath: string | undefined;
/** Whether to show progress bars. Good when running locally, bad when running on travis / azure. */
readonly progress: boolean;
/** Disabled on azure since it has problems logging errors from other processes. */
readonly parseInParallel: boolean;
}
export namespace Options {
/** Options for running locally. */
export const defaults: TesterOptions = { definitelyTypedPath: "../DefinitelyTyped", progress: true, parseInParallel: true };
export const azure: Options = { definitelyTypedPath: undefined, progress: false, parseInParallel: false };
}
export interface TesterOptions extends Options {
// Tester can only run on files stored on-disk.
readonly definitelyTypedPath: string;
}
export function readDataFile(generatedBy: string, fileName: string): Promise<object> {
return readFileAndWarn(generatedBy, dataFilePath(fileName));
}
/** If a file doesn't exist, warn and tell the step it should have been generated by. */
export async function readFileAndWarn(generatedBy: string, filePath: string): Promise<object> {
try {
return await readJson(filePath);
} catch (e) {
console.error(`Run ${generatedBy} first!`);
throw e;
}
}
export async function writeDataFile(filename: string, content: {}, formatted = true): Promise<void> {
await ensureDir(dataDirPath);
await writeJson(dataFilePath(filename), content, formatted);
}
export function dataFilePath(filename: string): string {
return joinPaths(dataDirPath, filename);
}
================================================
FILE: src/lib/definition-parser-worker.ts
================================================
import assert = require("assert");
import process = require("process");
import { getLocallyInstalledDefinitelyTyped } from "../get-definitely-typed";
import { logUncaughtErrors } from "../util/util";
import { getTypingInfo } from "./definition-parser";
// This file is "called" by runWithChildProcesses from parse-definition.ts
export const definitionParserWorkerFilename = __filename;
if (!module.parent) {
process.on("message", message => {
assert(process.argv.length === 3);
const typesPath = process.argv[2];
// tslint:disable-next-line no-async-without-await
logUncaughtErrors(async () => {
for (const packageName of message as ReadonlyArray<string>) {
const data = getTypingInfo(packageName, getLocallyInstalledDefinitelyTyped(typesPath).subDir(packageName));
process.send!({ data, packageName });
}
});
});
}
================================================
FILE: src/lib/definition-parser.test.ts
================================================
// tslint:disable:object-literal-key-quotes
import { createMockDT } from "../mocks";
import { getTypingInfo } from "./definition-parser";
describe(getTypingInfo, () => {
it("keys data by major.minor version", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "1.42");
dt.addOldVersionOfPackage("jquery", "2");
const info = getTypingInfo("jquery", dt.pkgFS("jquery"));
expect(Object.keys(info).sort()).toEqual(["1.42", "2.0", "3.3"]);
});
describe("concerning multiple versions", () => {
it("records what the version directory looks like on disk", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "2");
dt.addOldVersionOfPackage("jquery", "1.5");
const info = getTypingInfo("jquery", dt.pkgFS("jquery"));
expect(info).toEqual({
"1.5": expect.objectContaining({
libraryVersionDirectoryName: "1.5",
}),
"2.0": expect.objectContaining({
libraryVersionDirectoryName: "2",
}),
"3.3": expect.objectContaining({
// The latest version does not have its own version directory
libraryVersionDirectoryName: undefined,
}),
});
});
it("records a path mapping to the version directory", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "2");
dt.addOldVersionOfPackage("jquery", "1.5");
const info = getTypingInfo("jquery", dt.pkgFS("jquery"));
expect(info).toEqual({
"1.5": expect.objectContaining({
pathMappings: [{
packageName: "jquery",
version: { major: 1, minor: 5 },
}],
}),
"2.0": expect.objectContaining({
pathMappings: [{
packageName: "jquery",
version: { major: 2, minor: undefined },
}],
}),
"3.3": expect.objectContaining({
// The latest version does not have path mappings of its own
pathMappings: [],
}),
});
});
describe("validation thereof", () => {
it("throws if a directory exists for the latest major version", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "3");
expect(() => {
getTypingInfo("jquery", dt.pkgFS("jquery"));
}).toThrow(
"The latest version is 3.3, so the subdirectory 'v3' is not allowed; " +
"since it applies to any 3.* version, up to and including 3.3.",
);
});
it("throws if a directory exists for the latest minor version", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "3.3");
expect(() => {
getTypingInfo("jquery", dt.pkgFS("jquery"));
}).toThrow(
"The latest version is 3.3, so the subdirectory 'v3.3' is not allowed.",
);
});
it("does not throw when a minor version is older than the latest", () => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "3.0");
expect(() => {
getTypingInfo("jquery", dt.pkgFS("jquery"));
}).not.toThrow();
});
});
});
});
================================================
FILE: src/lib/definition-parser.ts
================================================
import { parseHeaderOrFail, TypeScriptVersion } from "definitelytyped-header-parser";
import * as ts from "typescript";
import { FS } from "../get-definitely-typed";
import {
computeHash, filter, flatMap, hasWindowsSlashes, join, mapDefined, sort, split, unique, unmangleScopedPackage, withoutStart,
} from "../util/util";
import { allReferencedFiles, createSourceFile, getModuleInfo, getTestDependencies } from "./module-info";
import {
formatTypingVersion,
getLicenseFromPackageJson,
PackageId,
PackageJsonDependency,
PathMapping,
TypingsDataRaw,
TypingsVersionsRaw,
TypingVersion,
} from "./packages";
import { dependenciesWhitelist } from "./settings";
function matchesVersion(typingsDataRaw: TypingsDataRaw, version: TypingVersion, considerLibraryMinorVersion: boolean) {
return typingsDataRaw.libraryMajorVersion === version.major
&& (considerLibraryMinorVersion ?
(version.minor === undefined || typingsDataRaw.libraryMinorVersion === version.minor)
: true);
}
function formattedLibraryVersion(typingsDataRaw: TypingsDataRaw) {
return `${typingsDataRaw.libraryMajorVersion}.${typingsDataRaw.libraryMinorVersion}`;
}
/** @param fs Rooted at the package's directory, e.g. `DefinitelyTyped/types/abs` */
export function getTypingInfo(packageName: string, fs: FS): TypingsVersionsRaw {
if (packageName !== packageName.toLowerCase()) {
throw new Error(`Package name \`${packageName}\` should be strictly lowercase`);
}
interface OlderVersionDir { readonly directoryName: string; readonly version: TypingVersion; }
const [rootDirectoryLs, olderVersionDirectories] = split<string, OlderVersionDir>(fs.readdir(), fileOrDirectoryName => {
const version = parseVersionFromDirectoryName(fileOrDirectoryName);
return version === undefined ? undefined : { directoryName: fileOrDirectoryName, version };
});
const considerLibraryMinorVersion = olderVersionDirectories.some(({ version }) => version.minor !== undefined);
const latestData: TypingsDataRaw = {
libraryVersionDirectoryName: undefined,
...combineDataForAllTypesVersions(packageName, rootDirectoryLs, fs, undefined),
};
const older = olderVersionDirectories.map(({ directoryName, version: directoryVersion }) => {
if (matchesVersion(latestData, directoryVersion, considerLibraryMinorVersion)) {
const latest = `${latestData.libraryMajorVersion}.${latestData.libraryMinorVersion}`;
throw new Error(
`The latest version is ${latest}, so the subdirectory '${directoryName}' is not allowed` +
(`v${latest}` === directoryName ?
"." : `; since it applies to any ${latestData.libraryMajorVersion}.* version, up to and including ${latest}.`),
);
}
const ls = fs.readdir(directoryName);
const data: TypingsDataRaw = {
libraryVersionDirectoryName: formatTypingVersion(directoryVersion),
...combineDataForAllTypesVersions(packageName, ls, fs.subDir(directoryName), directoryVersion),
};
if (!matchesVersion(data, directoryVersion, considerLibraryMinorVersion)) {
if (considerLibraryMinorVersion) {
throw new Error(
`Directory ${directoryName} indicates major.minor version ${directoryVersion.major}.${directoryVersion.minor}, ` +
`but header indicates major.minor version ${data.libraryMajorVersion}.${data.libraryMinorVersion}`,
);
}
throw new Error(
`Directory ${directoryName} indicates major version ${directoryVersion.major}, but header indicates major version ` +
data.libraryMajorVersion.toString(),
);
}
return data;
});
const res: TypingsVersionsRaw = {};
res[formattedLibraryVersion(latestData)] = latestData;
for (const o of older) {
res[formattedLibraryVersion(o)] = o;
}
return res;
}
const packageJsonName = "package.json";
interface LsMinusTypesVersionsAndPackageJson {
readonly remainingLs: ReadonlyArray<string>;
readonly typesVersions: ReadonlyArray<TypeScriptVersion>;
readonly hasPackageJson: boolean;
}
function getTypesVersionsAndPackageJson(ls: ReadonlyArray<string>): LsMinusTypesVersionsAndPackageJson {
const withoutPackageJson = ls.filter(name => name !== packageJsonName);
const [remainingLs, typesVersions] = split(withoutPackageJson, fileOrDirectoryName => {
const match = /^ts(\d+\.\d+)$/.exec(fileOrDirectoryName);
if (match === null) { return undefined; }
const version = match[1];
if (parseInt(version, 10) < 3) {
throw new Error(`Directory name starting with 'ts' should be a TypeScript version newer than 3.0. Got: ${version}`);
}
return version as TypeScriptVersion;
});
return { remainingLs, typesVersions, hasPackageJson: withoutPackageJson.length !== ls.length };
}
/**
* Parses a directory name into a version that either holds a single major version or a major and minor version.
*
* @example
*
* ```ts
* parseVersionFromDirectoryName("v1") // { major: 1 }
* parseVersionFromDirectoryName("v0.61") // { major: 0, minor: 61 }
* ```
*/
export function parseVersionFromDirectoryName(directoryName: string): TypingVersion | undefined {
const match = /^v(\d+)(\.(\d+))?$/.exec(directoryName);
if (match === null) {
return undefined;
}
return {
major: Number(match[1]),
minor: match[3] !== undefined ? Number(match[3]) : undefined, // tslint:disable-line strict-type-predicates (false positive)
};
}
function combineDataForAllTypesVersions(
typingsPackageName: string,
ls: ReadonlyArray<string>,
fs: FS,
directoryVersion: TypingVersion | undefined,
): Omit<TypingsDataRaw, "libraryVersionDirectoryName"> {
const { remainingLs, typesVersions, hasPackageJson } = getTypesVersionsAndPackageJson(ls);
// Every typesVersion has an index.d.ts, but only the root index.d.ts should have a header.
const { contributors, libraryMajorVersion, libraryMinorVersion, typeScriptVersion: minTsVersion, libraryName, projects } =
parseHeaderOrFail(readFileAndThrowOnBOM("index.d.ts", fs));
const dataForRoot = getTypingDataForSingleTypesVersion(undefined, typingsPackageName, fs.debugPath(), remainingLs, fs, directoryVersion);
const dataForOtherTypesVersions = typesVersions.map(tsVersion => {
const subFs = fs.subDir(`ts${tsVersion}`);
return getTypingDataForSingleTypesVersion(tsVersion, typingsPackageName, fs.debugPath(), subFs.readdir(), subFs, directoryVersion);
});
const allTypesVersions = [dataForRoot, ...dataForOtherTypesVersions];
const packageJson = hasPackageJson ? fs.readJson(packageJsonName) as { readonly license?: unknown, readonly dependencies?: unknown } : {};
const license = getLicenseFromPackageJson(packageJson.license);
const packageJsonDependencies = checkPackageJsonDependencies(packageJson.dependencies, packageJsonName);
const files = Array.from(flatMap(allTypesVersions, ({ typescriptVersion, declFiles }) =>
declFiles.map(file =>
typescriptVersion === undefined ? file : `ts${typescriptVersion}/${file}`)));
return {
libraryName,
typingsPackageName,
projectName: projects[0], // TODO: collect multiple project names
contributors,
libraryMajorVersion,
libraryMinorVersion,
minTsVersion,
typesVersions,
files,
license,
// TODO: Explicit type arguments shouldn't be necessary. https://github.com/Microsoft/TypeScript/issues/27507
dependencies: getAllUniqueValues<"dependencies", PackageId>(allTypesVersions, "dependencies"),
testDependencies: getAllUniqueValues<"testDependencies", string>(allTypesVersions, "testDependencies"),
pathMappings: getAllUniqueValues<"pathMappings", PathMapping>(allTypesVersions, "pathMappings"),
packageJsonDependencies,
contentHash: hash(hasPackageJson ? [...files, packageJsonName] : files, mapDefined(allTypesVersions, a => a.tsconfigPathsForHash), fs),
globals: getAllUniqueValues<"globals", string>(allTypesVersions, "globals"),
declaredModules: getAllUniqueValues<"declaredModules", string>(allTypesVersions, "declaredModules"),
};
}
function getAllUniqueValues<K extends string, T>(records: ReadonlyArray<Record<K, ReadonlyArray<T>>>, key: K): ReadonlyArray<T> {
return unique(flatMap(records, x => x[key]));
}
interface TypingDataFromIndividualTypeScriptVersion {
/** Undefined for root (which uses `// TypeScript Version: ` comment instead) */
readonly typescriptVersion: TypeScriptVersion | undefined;
readonly dependencies: ReadonlyArray<PackageId>;
readonly testDependencies: ReadonlyArray<string>;
readonly pathMappings: ReadonlyArray<PathMapping>;
readonly declFiles: ReadonlyArray<string>;
readonly tsconfigPathsForHash: string | undefined;
readonly globals: ReadonlyArray<string>;
readonly declaredModules: ReadonlyArray<string>;
}
/**
* @param typescriptVersion Set if this is in e.g. a `ts3.1` directory.
* @param packageName Name of the outermost directory; e.g. for "node/v4" this is just "node".
* @param ls All file/directory names in `directory`.
* @param fs FS rooted at the directory for this particular TS version, e.g. `types/abs/ts3.1` or `types/abs` when typescriptVersion is undefined.
*/
function getTypingDataForSingleTypesVersion(
typescriptVersion: TypeScriptVersion | undefined,
packageName: string,
packageDirectory: string,
ls: ReadonlyArray<string>,
fs: FS,
directoryVersion: TypingVersion | undefined,
): TypingDataFromIndividualTypeScriptVersion {
const tsconfig = fs.readJson("tsconfig.json") as TsConfig;
checkFilesFromTsConfig(packageName, tsconfig, fs.debugPath());
const { types, tests } = allReferencedFiles(tsconfig.files!, fs, packageName, packageDirectory);
const usedFiles = new Set([...types.keys(), ...tests.keys(), "tsconfig.json", "tslint.json"]);
const otherFiles = ls.indexOf(unusedFilesName) > -1 ? (fs.readFile(unusedFilesName)).split(/\r?\n/g).filter(Boolean) : [];
checkAllFilesUsed(ls, usedFiles, otherFiles, packageName, fs);
for (const untestedTypeFile of filter(otherFiles, name => name.endsWith(".d.ts"))) {
// add d.ts files from OTHER_FILES.txt in order get their dependencies
types.set(untestedTypeFile, createSourceFile(untestedTypeFile, fs.readFile(untestedTypeFile)));
}
const { dependencies: dependenciesWithDeclaredModules, globals, declaredModules } = getModuleInfo(packageName, types);
const declaredModulesSet = new Set(declaredModules);
// Don't count an import of "x" as a dependency if we saw `declare module "x"` somewhere.
const dependenciesSet = new Set(filter(dependenciesWithDeclaredModules, m => !declaredModulesSet.has(m)));
const testDependencies = Array.from(
filter(
getTestDependencies(packageName, types, tests.keys(), dependenciesSet, fs),
m => !declaredModulesSet.has(m),
),
);
const { dependencies, pathMappings } = calculateDependencies(packageName, tsconfig, dependenciesSet, directoryVersion);
const tsconfigPathsForHash = JSON.stringify(tsconfig.compilerOptions.paths);
return {
typescriptVersion,
dependencies,
testDependencies,
pathMappings,
globals,
declaredModules,
declFiles: sort(types.keys()),
tsconfigPathsForHash,
};
}
function checkPackageJsonDependencies(dependencies: unknown, path: string): ReadonlyArray<PackageJsonDependency> {
if (dependencies === undefined) { // tslint:disable-line strict-type-predicates (false positive)
return [];
}
if (dependencies === null || typeof dependencies !== "object") { // tslint:disable-line strict-type-predicates
throw new Error(`${path} should contain "dependencies" or not exist.`);
}
const deps: PackageJsonDependency[] = [];
for (const dependencyName of Object.keys(dependencies!)) { // `dependencies` cannot be null because of check above.
if (!dependenciesWhitelist.has(dependencyName)) {
const msg = dependencyName.startsWith("@types/")
? `Dependency ${dependencyName} not in whitelist.
Don't use a 'package.json' for @types dependencies unless this package relies on
an old version of types that have since been moved to the source repo.
For example, if package *P* used to have types on Definitely Typed at @types/P,
but now has its own types, a dependent package *D* will need to use package.json
to refer to @types/P if it relies on old versions of P's types.
In this case, please make a pull request to types-publisher adding @types/P to \`dependenciesWhitelist.txt\`.`
: `Dependency ${dependencyName} not in whitelist.
If you are depending on another \`@types\` package, do *not* add it to a \`package.json\`. Path mapping should make the import work.
For namespaced dependencies you then have to add a \`paths\` mapping from \`@namespace/library\` to \`namespace__library\` in \`tsconfig.json\`.
If this is an external library that provides typings, please make a pull request to types-publisher adding it to \`dependenciesWhitelist.txt\`.`;
throw new Error(`In ${path}: ${msg}`);
}
const version = (dependencies as { [key: string]: unknown })[dependencyName];
if (typeof version !== "string") { // tslint:disable-line strict-type-predicates
throw new Error(`In ${path}: Dependency version for ${dependencyName} should be a string.`);
}
deps.push({ name: dependencyName, version });
}
return deps;
}
function checkFilesFromTsConfig(packageName: string, tsconfig: TsConfig, directoryPath: string): void {
const tsconfigPath = `${directoryPath}/tsconfig.json`;
if (tsconfig.include) {
throw new Error(`In tsconfig, don't use "include", must use "files"`);
}
const files = tsconfig.files;
if (!files) {
throw new Error(`${tsconfigPath} needs to specify "files"`);
}
for (const file of files) {
if (file.startsWith("./")) {
throw new Error(`In ${tsconfigPath}: Unnecessary "./" at the start of ${file}`);
}
if (file.endsWith(".d.ts") && file !== "index.d.ts") {
throw new Error(`${packageName}: Only index.d.ts may be listed explicitly in tsconfig's "files" entry.
Other d.ts files must either be referenced through index.d.ts, tests, or added to OTHER_FILES.txt.`);
}
if (!file.endsWith(".d.ts") && !file.startsWith("test/")) {
const expectedName = `${packageName}-tests.ts`;
if (file !== expectedName && file !== `${expectedName}x`) {
const message = file.endsWith(".ts") || file.endsWith(".tsx")
? `Expected file '${file}' to be named '${expectedName}' or to be inside a '${directoryPath}/test/' directory`
: (`Unexpected file extension for '${file}' -- expected '.ts' or '.tsx' (maybe this should not be in "files", but ` +
"OTHER_FILES.txt)");
throw new Error(message);
}
}
}
}
interface TsConfig {
include?: ReadonlyArray<string>;
files?: ReadonlyArray<string>;
compilerOptions: ts.CompilerOptions;
}
/** In addition to dependencies found in source code, also get dependencies from tsconfig. */
interface DependenciesAndPathMappings { readonly dependencies: ReadonlyArray<PackageId>; readonly pathMappings: ReadonlyArray<PathMapping>; }
function calculateDependencies(
packageName: string,
tsconfig: TsConfig,
dependencyNames: ReadonlySet<string>,
directoryVersion: TypingVersion | undefined,
): DependenciesAndPathMappings {
const paths = tsconfig.compilerOptions && tsconfig.compilerOptions.paths || {};
const dependencies: PackageId[] = [];
const pathMappings: PathMapping[] = [];
for (const dependencyName of Object.keys(paths)) {
// Might have a path mapping for "foo/*" to support subdirectories
const rootDirectory = withoutEnd(dependencyName, "/*");
if (rootDirectory !== undefined) {
if (!(rootDirectory in paths)) {
throw new Error(`In ${packageName}: found path mapping for ${dependencyName} but not for ${rootDirectory}`);
}
continue;
}
const pathMappingList = paths[dependencyName];
if (pathMappingList.length !== 1) {
throw new Error(`In ${packageName}: Path mapping for ${dependencyName} may only have 1 entry.`);
}
const pathMapping = pathMappingList[0];
// Path mapping may be for "@foo/bar" -> "foo__bar".
const scopedPackageName = unmangleScopedPackage(pathMapping);
if (scopedPackageName !== undefined) {
if (dependencyName !== scopedPackageName) {
throw new Error(`Expected directory ${pathMapping} to be the path mapping for ${dependencyName}`);
}
continue;
}
const pathMappingVersion = parseDependencyVersionFromPath(dependencyName, dependencyName, pathMapping);
if (dependencyName === packageName) {
if (directoryVersion === undefined) {
throw new Error(`In ${packageName}: Latest version of a package should not have a path mapping for itself.`);
}
if (
directoryVersion.major !== pathMappingVersion.major
|| directoryVersion.minor !== pathMappingVersion.minor
) {
const correctPathMapping = [`${dependencyName}/v${formatTypingVersion(directoryVersion)}`];
throw new Error(`In ${packageName}: Must have a "paths" entry of "${dependencyName}": ${JSON.stringify(correctPathMapping)}`);
}
} else {
if (dependencyNames.has(dependencyName)) {
dependencies.push({ name: dependencyName, version: pathMappingVersion });
}
}
// Else, the path mapping may be necessary if it is for a transitive dependency. We will check this in check-parse-results.
pathMappings.push({ packageName: dependencyName, version: pathMappingVersion });
}
if (directoryVersion !== undefined && !(paths && packageName in paths)) {
const mapping = JSON.stringify([`${packageName}/v${formatTypingVersion(directoryVersion)}`]);
throw new Error(
`${packageName}: Older version ${formatTypingVersion(directoryVersion)} must have a "paths" entry of "${packageName}": ${mapping}`,
);
}
for (const dependency of dependencyNames) {
if (!dependencies.some(d => d.name === dependency) && !nodeBuiltins.has(dependency)) {
dependencies.push({ name: dependency, version: "*" });
}
}
return { dependencies, pathMappings };
}
const nodeBuiltins: ReadonlySet<string> = new Set([
"assert", "async_hooks", "buffer", "child_process", "cluster", "console", "constants", "crypto",
"dgram", "dns", "domain", "events", "fs", "http", "http2", "https", "module", "net", "os",
"path", "perf_hooks", "process", "punycode", "querystring", "readline", "repl", "stream",
"string_decoder", "timers", "tls", "tty", "url", "util", "v8", "vm", "zlib",
]);
function parseDependencyVersionFromPath(packageName: string, dependencyName: string, dependencyPath: string): TypingVersion {
const versionString = withoutStart(dependencyPath, `${dependencyName}/`);
const version = versionString === undefined ? undefined : parseVersionFromDirectoryName(versionString);
if (version === undefined) {
throw new Error(`In ${packageName}, unexpected path mapping for ${dependencyName}: '${dependencyPath}'`);
}
return version;
}
function withoutEnd(s: string, end: string): string | undefined {
if (s.endsWith(end)) {
return s.slice(0, s.length - end.length);
}
return undefined;
}
function hash(files: ReadonlyArray<string>, tsconfigPathsForHash: ReadonlyArray<string>, fs: FS): string {
const fileContents = files.map(f => `${f}**${readFileAndThrowOnBOM(f, fs)}`);
let allContent = fileContents.join("||");
for (const path of tsconfigPathsForHash) {
allContent += path;
}
return computeHash(allContent);
}
export function readFileAndThrowOnBOM(fileName: string, fs: FS): string {
const text = fs.readFile(fileName);
if (text.charCodeAt(0) === 0xFEFF) {
const commands = [
"npm install -g strip-bom-cli",
`strip-bom ${fileName} > fix`,
`mv fix ${fileName}`,
];
throw new Error(`File '${fileName}' has a BOM. Try using:\n${commands.join("\n")}`);
}
return text;
}
const unusedFilesName = "OTHER_FILES.txt";
function checkAllFilesUsed(ls: ReadonlyArray<string>, usedFiles: Set<string>, otherFiles: string[], packageName: string, fs: FS): void {
// Double-check that no windows "\\" broke in.
for (const fileName of usedFiles) {
if (hasWindowsSlashes(fileName)) {
throw new Error(`In ${packageName}: windows slash detected in ${fileName}`);
}
}
checkAllUsedRecur(new Set(ls), usedFiles, new Set(otherFiles), fs);
}
function checkAllUsedRecur(ls: Iterable<string>, usedFiles: Set<string>, unusedFiles: Set<string>, fs: FS): void {
for (const lsEntry of ls) {
if (usedFiles.has(lsEntry)) {
continue;
}
if (unusedFiles.has(lsEntry)) {
unusedFiles.delete(lsEntry);
continue;
}
if (fs.isDirectory(lsEntry)) {
const subdir = fs.subDir(lsEntry);
// We allow a "scripts" directory to be used for scripts.
if (lsEntry === "node_modules" || lsEntry === "scripts") {
continue;
}
const lssubdir = subdir.readdir();
if (lssubdir.length === 0) {
// tslint:disable-next-line strict-string-expressions
throw new Error(`Empty directory ${subdir} (${join(usedFiles)})`);
}
function takeSubdirectoryOutOfSet(originalSet: Set<string>): Set<string> {
const subdirSet = new Set<string>();
for (const file of originalSet) {
const sub = withoutStart(file, `${lsEntry}/`);
if (sub !== undefined) {
originalSet.delete(file);
subdirSet.add(sub);
}
}
return subdirSet;
}
checkAllUsedRecur(lssubdir, takeSubdirectoryOutOfSet(usedFiles), takeSubdirectoryOutOfSet(unusedFiles), subdir);
} else {
if (lsEntry.toLowerCase() !== "readme.md" && lsEntry !== "NOTICE" && lsEntry !== ".editorconfig" && lsEntry !== unusedFilesName) {
throw new Error(`Unused file ${fs.debugPath()}/${lsEntry} (used files: ${JSON.stringify(Array.from(usedFiles))})`);
}
}
}
for (const unusedFile of unusedFiles) {
if (usedFiles.has(unusedFile)) {
throw new Error(`File ${fs.debugPath()}/${unusedFile} listed in ${unusedFilesName} is already reachable from tsconfig.json.`);
}
throw new Error(`File ${fs.debugPath()}/${unusedFile} listed in ${unusedFilesName} does not exist.`);
}
}
================================================
FILE: src/lib/module-info.test.ts
================================================
import * as ts from "typescript";
import { Dir, InMemoryDT } from "../get-definitely-typed";
import { createMockDT } from "../mocks";
import { testo } from "../util/test";
import { allReferencedFiles, getModuleInfo, getTestDependencies } from "./module-info";
const fs = createMockDT().fs;
function getBoringReferences() {
return allReferencedFiles(["index.d.ts", "boring-tests.ts"], fs.subDir("types").subDir("boring"), "boring", "types/boring");
}
testo({
allReferencedFilesFromTsconfigFiles() {
const { types, tests } = getBoringReferences();
expect(Array.from(types.keys())).toEqual(["index.d.ts", "secondary.d.ts", "quaternary.d.ts", "tertiary.d.ts", "commonjs.d.ts", "v1.d.ts"]);
expect(Array.from(tests.keys())).toEqual(["boring-tests.ts"]);
},
allReferencedFilesFromTestIncludesSecondaryInternalFiles() {
const { types, tests } = allReferencedFiles(["boring-tests.ts"], fs.subDir("types").subDir("boring"), "boring", "types/boring");
expect(Array.from(types.keys())).toEqual(["secondary.d.ts", "quaternary.d.ts", "tertiary.d.ts", "commonjs.d.ts", "v1.d.ts"]);
expect(Array.from(tests.keys())).toEqual(["boring-tests.ts"]);
},
allReferencedFilesFromTsconfigGlobal() {
const { types, tests } = allReferencedFiles(["jquery-tests.ts", "index.d.ts"], fs.subDir("types").subDir("jquery"), "jquery", "types/jquery");
expect(Array.from(types.keys())).toEqual(["index.d.ts", "JQuery.d.ts"]);
expect(Array.from(tests.keys())).toEqual(["jquery-tests.ts"]);
},
allReferencedFilesFromTestIncludesSecondaryTripleSlashTypes() {
const { types, tests } = allReferencedFiles(
["globby-tests.ts", "test/other-tests.ts"],
fs.subDir("types").subDir("globby"),
"globby",
"types/globby",
);
expect(Array.from(types.keys())).toEqual(["merges.d.ts"]);
expect(Array.from(tests.keys())).toEqual(["globby-tests.ts", "test/other-tests.ts"]);
},
getModuleInfoWorksWithOtherFiles() {
const { types } = getBoringReferences();
// written as if it were from OTHER_FILES.txt
types.set(
"untested.d.ts",
ts.createSourceFile("untested.d.ts", fs.subDir("types").subDir("boring").readFile("untested.d.ts"), ts.ScriptTarget.Latest, false),
);
const i = getModuleInfo("boring", types);
expect(i.dependencies).toEqual(new Set(["manual", "react", "react-default", "things", "vorticon"]));
},
getModuleInfoForNestedTypeReferences() {
const { types } = allReferencedFiles(
["index.d.ts", "globby-tests.ts", "test/other-tests.ts"],
fs.subDir("types").subDir("globby"),
"globby",
"types/globby",
);
expect(Array.from(types.keys())).toEqual(["index.d.ts", "sneaky.d.ts", "merges.d.ts"]);
const i = getModuleInfo("globby", types);
expect(i.dependencies).toEqual(new Set(["andere"]));
},
versionTypeRefThrows() {
const fail = new Dir(undefined);
const memFS = new InMemoryDT(fail, "typeref-fails");
fail.set("index.d.ts", `// Type definitions for fail 1.0
// Project: https://youtube.com/typeref-fails
// Definitions by: Type Ref Fails <https://github.com/typeref-fails>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
/// <reference types="elser/v3" />
`);
const { types } = allReferencedFiles(["index.d.ts"], memFS, "typeref-fails", "types/typeref-fails");
expect(Array.from(types.keys())).toEqual(["index.d.ts"]);
expect(() => getModuleInfo("typeref-fails", types)).toThrow("do not directly import specific versions of another types package");
},
selfVersionTypeRefAllowed() {
const fail = new Dir(undefined);
const memFS = new InMemoryDT(fail, "typeref-fails");
fail.set("index.d.ts", `// Type definitions for fail 1.0
// Project: https://youtube.com/typeref-fails
// Definitions by: Type Ref Fails <https://github.com/typeref-fails>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
/// <reference types="fail/v3" />
`);
const { types } = allReferencedFiles(["index.d.ts"], memFS, "typeref-fails", "types/typeref-fails");
expect(Array.from(types.keys())).toEqual(["index.d.ts"]);
const i = getModuleInfo("fail", types);
expect(i.dependencies).toEqual(new Set([]));
},
getTestDependenciesWorks() {
const { types, tests } = getBoringReferences();
const i = getModuleInfo("boring", types);
const d = getTestDependencies("boring", types, tests.keys(), i.dependencies, fs.subDir("types").subDir("boring"));
expect(d).toEqual(new Set(["super-big-fun-hus"]));
},
});
================================================
FILE: src/lib/module-info.ts
================================================
import assert = require("assert");
import * as path from "path";
import * as ts from "typescript";
import { FS } from "../get-definitely-typed";
import { hasWindowsSlashes, joinPaths, normalizeSlashes, sort } from "../util/util";
import { readFileAndThrowOnBOM } from "./definition-parser";
export function getModuleInfo(packageName: string, all: Map<string, ts.SourceFile>): ModuleInfo {
const dependencies = new Set<string>();
const declaredModules: string[] = [];
const globals = new Set<string>();
function addDependency(ref: string): void {
if (ref.startsWith(".")) { return; }
const dependency = rootName(ref, all, packageName);
if (dependency !== packageName) {
dependencies.add(dependency);
}
// TODO: else throw new Error(`Package ${packageName} references itself. (via ${src.fileName})`);
}
for (const sourceFile of all.values()) {
for (const ref of imports(sourceFile)) {
addDependency(ref);
}
for (const ref of sourceFile.typeReferenceDirectives) {
addDependency(ref.fileName);
}
if (ts.isExternalModule(sourceFile)) {
if (sourceFileExportsSomething(sourceFile)) {
declaredModules.push(properModuleName(packageName, sourceFile.fileName));
const namespaceExport = sourceFile.statements.find(ts.isNamespaceExportDeclaration);
if (namespaceExport) {
globals.add(namespaceExport.name.text);
}
}
} else {
for (const node of sourceFile.statements) {
switch (node.kind) {
case ts.SyntaxKind.ModuleDeclaration: {
const decl = node as ts.ModuleDeclaration;
const name = decl.name.text;
if (decl.name.kind === ts.SyntaxKind.StringLiteral) {
declaredModules.push(assertNoWindowsSlashes(packageName, name));
} else if (isValueNamespace(decl)) {
globals.add(name);
}
break;
}
case ts.SyntaxKind.VariableStatement:
for (const decl of (node as ts.VariableStatement).declarationList.declarations) {
if (decl.name.kind === ts.SyntaxKind.Identifier) {
globals.add(decl.name.text);
}
}
break;
case ts.SyntaxKind.EnumDeclaration:
case ts.SyntaxKind.ClassDeclaration:
case ts.SyntaxKind.FunctionDeclaration: {
// Deliberately not doing this for types, because those won't show up in JS code and can't be used for ATA
const nameNode = (node as ts.EnumDeclaration | ts.ClassDeclaration | ts.FunctionDeclaration).name;
if (nameNode) {
globals.add(nameNode.text);
}
break;
}
case ts.SyntaxKind.ImportEqualsDeclaration:
case ts.SyntaxKind.InterfaceDeclaration:
case ts.SyntaxKind.TypeAliasDeclaration:
break;
default:
throw new Error(`Unexpected node kind ${ts.SyntaxKind[node.kind]}`);
}
}
}
}
return { dependencies, declaredModules, globals: sort(globals) };
}
/**
* A file is a proper module if it is an external module *and* it has at least one export.
* A module with only imports is not a proper module; it likely just augments some other module.
*/
function sourceFileExportsSomething({ statements }: ts.SourceFile): boolean {
return statements.some(statement => {
switch (statement.kind) {
case ts.SyntaxKind.ImportEqualsDeclaration:
case ts.SyntaxKind.ImportDeclaration:
return false;
case ts.SyntaxKind.ModuleDeclaration:
return (statement as ts.ModuleDeclaration).name.kind === ts.SyntaxKind.Identifier;
default:
return true;
}
});
}
interface ModuleInfo {
dependencies: Set<string>;
// Anything from a `declare module "foo"`
declaredModules: string[];
// Every global symbol
globals: string[];
}
/**
* Given a file name, get the name of the module it declares.
* `foo/index.d.ts` declares "foo", `foo/bar.d.ts` declares "foo/bar", "foo/bar/index.d.ts" declares "foo/bar"
*/
function properModuleName(folderName: string, fileName: string): string {
const part = path.basename(fileName) === "index.d.ts" ? path.dirname(fileName) : withoutExtension(fileName, ".d.ts");
return part === "." ? folderName : joinPaths(folderName, part);
}
/**
* "foo/bar/baz" -> "foo"; "@foo/bar/baz" -> "@foo/bar"
* Note: Throws an error for references like
* "bar/v3" because referencing old versions of *other* packages is illegal;
* those directories won't exist in the published @types package.
*/
function rootName(importText: string, typeFiles: Map<string, unknown>, packageName: string): string {
let slash = importText.indexOf("/");
// Root of `@foo/bar/baz` is `@foo/bar`
if (importText.startsWith("@")) {
// Use second "/"
slash = importText.indexOf("/", slash + 1);
}
const root = importText.slice(0, slash);
const postImport = importText.slice(slash + 1);
if (slash > -1 && postImport.match(/v\d+$/) && !typeFiles.has(postImport + ".d.ts") && root !== packageName) {
throw new Error(`${importText}: do not directly import specific versions of another types package.
You should work with the latest version of ${root} instead.`);
}
return slash === -1 ? importText : root;
}
function withoutExtension(str: string, ext: string): string {
assert(str.endsWith(ext));
return str.slice(0, str.length - ext.length);
}
/** Returns a map from filename (path relative to `directory`) to the SourceFile we parsed for it. */
export function allReferencedFiles(
entryFilenames: ReadonlyArray<string>, fs: FS, packageName: string, baseDirectory: string,
): { types: Map<string, ts.SourceFile>, tests: Map<string, ts.SourceFile> } {
const seenReferences = new Set<string>();
const types = new Map<string, ts.SourceFile>();
const tests = new Map<string, ts.SourceFile>();
entryFilenames.forEach(text => recur({ text, exact: true }));
return { types, tests };
function recur({ text, exact }: Reference): void {
if (seenReferences.has(text)) {
return;
}
seenReferences.add(text);
const resolvedFilename = exact ? text : resolveModule(text, fs);
if (fs.exists(resolvedFilename)) {
const src = createSourceFile(resolvedFilename, readFileAndThrowOnBOM(resolvedFilename, fs));
if (resolvedFilename.endsWith(".d.ts")) {
types.set(resolvedFilename, src);
} else {
tests.set(resolvedFilename, src);
}
const refs = findReferencedFiles(
src,
packageName,
path.dirname(resolvedFilename),
normalizeSlashes(path.relative(baseDirectory, fs.debugPath())),
);
refs.forEach(recur);
}
}
}
function resolveModule(importSpecifier: string, fs: FS): string {
importSpecifier = importSpecifier.endsWith("/") ? importSpecifier.slice(0, importSpecifier.length - 1) : importSpecifier;
if (importSpecifier !== "." && importSpecifier !== "..") {
if (fs.exists(importSpecifier + ".d.ts")) {
return importSpecifier + ".d.ts";
}
if (fs.exists(importSpecifier + ".ts")) {
return importSpecifier + ".ts";
}
if (fs.exists(importSpecifier + ".tsx")) {
return importSpecifier + ".tsx";
}
}
return importSpecifier === "." ? "index.d.ts" : joinPaths(importSpecifier, "index.d.ts");
}
interface Reference {
/** <reference path> includes exact filename, so true. import "foo" may reference "foo.d.ts" or "foo/index.d.ts", so false. */
readonly exact: boolean;
text: string;
}
/**
* @param subDirectory The specific directory within the DefinitelyTyped directory we are in.
* For example, `baseDirectory` may be `react-router` and `subDirectory` may be `react-router/lib`.
* versionsBaseDirectory may be "" when not in typesVersions or ".." when inside `react-router/ts3.1`
*/
function findReferencedFiles(src: ts.SourceFile, packageName: string, subDirectory: string, baseDirectory: string) {
const refs: Reference[] = [];
for (const ref of src.referencedFiles) {
// Any <reference path="foo"> is assumed to be local
addReference({ text: ref.fileName, exact: true });
}
for (const ref of src.typeReferenceDirectives) {
// only <reference types="../packagename/x" /> references are local (or "packagename/x", though in 3.7 that doesn't work in DT).
if (ref.fileName.startsWith("../" + packageName + "/")) {
addReference({ text: ref.fileName, exact: false });
} else if (ref.fileName.startsWith(packageName + "/")) {
addReference({ text: convertToRelativeReference(ref.fileName), exact: false });
}
}
for (const ref of imports(src)) {
if (ref.startsWith(".")) {
addReference({ text: ref, exact: false });
}
if (ref.startsWith(packageName + "/")) {
addReference({ text: convertToRelativeReference(ref), exact: false });
}
}
return refs;
function addReference(ref: Reference): void {
// `path.normalize` may add windows slashes
const full = normalizeSlashes(path.normalize(joinPaths(subDirectory, assertNoWindowsSlashes(src.fileName, ref.text))));
// allow files in typesVersions directories (i.e. 'ts3.1') to reference files in parent directory
if (full.startsWith("../" + packageName + "/")) {
ref.text = full.slice(packageName.length + 4);
refs.push(ref);
return;
}
if (full.startsWith("..")
&& (baseDirectory === "" || path.normalize(joinPaths(baseDirectory, full)).startsWith(".."))) {
throw new Error(
`${src.fileName}: ` +
'Definitions must use global references to other packages, not parent ("../xxx") references.' +
`(Based on reference '${ref.text}')`);
}
ref.text = full;
refs.push(ref);
}
/** boring/foo -> ./foo when subDirectory === '.'; ../foo when it's === 'x'; ../../foo when it's 'x/y' */
function convertToRelativeReference(name: string) {
const relative = "." + "/..".repeat(subDirectory === "." ? 0 : subDirectory.split("/").length);
return relative + name.slice(packageName.length);
}
}
/**
* All strings referenced in `import` statements.
* Does *not* include <reference> directives.
*/
function* imports({ statements }: ts.SourceFile | ts.ModuleBlock): Iterable<string> {
for (const node of statements) {
switch (node.kind) {
case ts.SyntaxKind.ImportDeclaration:
case ts.SyntaxKind.ExportDeclaration: {
const { moduleSpecifier } = node as ts.ImportDeclaration | ts.ExportDeclaration;
if (moduleSpecifier && moduleSpecifier.kind === ts.SyntaxKind.StringLiteral) {
yield (moduleSpecifier as ts.StringLiteral).text;
}
break;
}
case ts.SyntaxKind.ImportEqualsDeclaration: {
const { moduleReference } = node as ts.ImportEqualsDeclaration;
if (moduleReference.kind === ts.SyntaxKind.ExternalModuleReference) {
yield parseRequire(moduleReference);
}
break;
}
case ts.SyntaxKind.ModuleDeclaration: {
const { name, body } = node as ts.ModuleDeclaration;
if (name.kind === ts.SyntaxKind.StringLiteral && body) {
yield* imports(body as ts.ModuleBlock);
}
break;
}
default:
}
}
}
function parseRequire(reference: ts.ExternalModuleReference): string {
const { expression } = reference;
if (!expression || !ts.isStringLiteral(expression)) {
throw new Error(`Bad 'import =' reference: ${reference.getText()}`);
}
return expression.text;
}
function isValueNamespace(ns: ts.ModuleDeclaration): boolean {
if (!ns.body) {
throw new Error("@types should not use shorthand ambient modules");
}
return ns.body.kind === ts.SyntaxKind.ModuleDeclaration
? isValueNamespace(ns.body as ts.ModuleDeclaration)
: (ns.body as ts.ModuleBlock).statements.some(statementDeclaresValue);
}
function statementDeclaresValue(statement: ts.Statement): boolean {
switch (statement.kind) {
case ts.SyntaxKind.VariableStatement:
case ts.SyntaxKind.ClassDeclaration:
case ts.SyntaxKind.FunctionDeclaration:
case ts.SyntaxKind.EnumDeclaration:
return true;
case ts.SyntaxKind.ModuleDeclaration:
return isValueNamespace(statement as ts.ModuleDeclaration);
case ts.SyntaxKind.InterfaceDeclaration:
case ts.SyntaxKind.TypeAliasDeclaration:
case ts.SyntaxKind.ImportEqualsDeclaration:
return false;
default:
throw new Error(`Forgot to implement ambient namespace statement ${ts.SyntaxKind[statement.kind]}`);
}
}
function assertNoWindowsSlashes(packageName: string, fileName: string): string {
if (hasWindowsSlashes(fileName)) {
throw new Error(`In ${packageName}: Use forward slash instead when referencing ${fileName}`);
}
return fileName;
}
export function getTestDependencies(
packageName: string,
typeFiles: Map<string, unknown>,
testFiles: Iterable<string>,
dependencies: ReadonlySet<string>,
fs: FS,
): Iterable<string> {
const testDependencies = new Set<string>();
for (const filename of testFiles) {
const content = readFileAndThrowOnBOM(filename, fs);
const sourceFile = createSourceFile(filename, content);
const { fileName, referencedFiles, typeReferenceDirectives } = sourceFile;
const filePath = () => path.join(packageName, fileName);
let hasImports = false;
let isModule = false;
let referencesSelf = false;
for (const { fileName: ref } of referencedFiles) {
throw new Error(`Test files should not use '<reference path="" />'. '${filePath()}' references '${ref}'.`);
}
for (const { fileName: referencedPackage } of typeReferenceDirectives) {
if (dependencies.has(referencedPackage)) {
throw new Error(
`'${filePath()}' unnecessarily references '${referencedPackage}', which is already referenced in the type definition.`);
}
if (referencedPackage === packageName) {
referencesSelf = true;
}
testDependencies.add(referencedPackage);
}
for (const imported of imports(sourceFile)) {
hasImports = true;
if (!imported.startsWith(".")) {
const dep = rootName(imported, typeFiles, packageName);
if (!dependencies.has(dep) && dep !== packageName) {
testDependencies.add(dep);
}
}
}
isModule = hasImports || (() => {
// FIXME: This results in files without imports to be walked twice,
// once in the `imports(...)` function, and once more here:
for (const node of sourceFile.statements) {
if (
node.kind === ts.SyntaxKind.ExportAssignment ||
node.kind === ts.SyntaxKind.ExportDeclaration
) {
return true;
}
}
return false;
})();
if (isModule && referencesSelf) {
throw new Error(`'${filePath()}' unnecessarily references the package. This can be removed.`);
}
}
return testDependencies;
}
export function createSourceFile(filename: string, content: string): ts.SourceFile {
return ts.createSourceFile(filename, content, ts.ScriptTarget.Latest, /*setParentNodes*/false);
}
================================================
FILE: src/lib/npm-client.ts
================================================
import assert = require("assert");
import { ensureFile, pathExists } from "fs-extra";
import RegClient = require("npm-registry-client");
import { resolve as resolveUrl } from "url";
import { Fetcher, readFile, readJson, sleep, writeJson } from "../util/io";
import { Logger, loggerWithErrors } from "../util/logging";
import { createTgz } from "../util/tgz";
import { assertNever, identity, joinPaths, mapToRecord, recordToMap } from "../util/util";
import { Registry } from "./common";
import { getSecret, Secret } from "./secrets";
import { githubRegistry, npmApi, npmRegistry, npmRegistryHostName } from "./settings";
const cacheFile = joinPaths(__dirname, "..", "..", "cache", "npmInfo.json");
export type NpmInfoCache = ReadonlyMap<string, NpmInfo>;
export interface NpmInfoRaw {
readonly "dist-tags": {
readonly [tag: string]: string;
};
readonly versions: NpmInfoRawVersions;
readonly time: {
readonly [s: string]: string;
};
}
export interface NpmInfoRawVersions {
readonly [version: string]: NpmInfoVersion;
}
// Processed npm info. Intentially kept small so it can be cached.
export interface NpmInfo {
readonly distTags: Map<string, string>;
readonly versions: Map<string, NpmInfoVersion>;
readonly time: Map<string, string>;
}
export interface NpmInfoVersion {
readonly typesPublisherContentHash?: string;
readonly deprecated?: string;
}
export interface CachedNpmInfoClient {
getNpmInfoFromCache(escapedPackageName: string): NpmInfo | undefined;
fetchAndCacheNpmInfo(escapedPackageName: string): Promise<NpmInfo | undefined>;
}
export async function withNpmCache<T>(uncachedClient: UncachedNpmInfoClient, cb: (client: CachedNpmInfoClient) => Promise<T>): Promise<T> {
const log = loggerWithErrors()[0];
let unroll: Map<string, NpmInfo>;
log.info(`Checking for cache file at ${cacheFile}...`);
const cacheFileExists = await pathExists(cacheFile);
if (cacheFileExists) {
log.info("Reading cache file...");
const cachedJson = await readJson(cacheFile) as Record<string, NpmInfoRaw>;
log.info(`Cache file ${cacheFile} exists, copying to map...`);
unroll = recordToMap(cachedJson, npmInfoFromJson);
} else {
log.info("Cache file doesn't exist, using empty map.");
unroll = new Map();
}
const res = await cb({ getNpmInfoFromCache, fetchAndCacheNpmInfo });
log.info("Writing npm cache.");
await ensureFile(cacheFile);
await writeJson(cacheFile, mapToRecord(unroll, jsonFromNpmInfo));
return res;
/** May return old info -- caller should check that this looks up-to-date. */
function getNpmInfoFromCache(escapedPackageName: string): NpmInfo | undefined {
return unroll.get(escapedPackageName);
}
/** Call this when the result of getNpmInfoFromCache looks potentially out-of-date. */
async function fetchAndCacheNpmInfo(escapedPackageName: string): Promise<NpmInfo | undefined> {
const info = await uncachedClient.fetchNpmInfo(escapedPackageName);
if (info) { unroll.set(escapedPackageName, info); }
return info;
}
}
export class UncachedNpmInfoClient {
private readonly fetcher = new Fetcher();
async fetchNpmInfo(escapedPackageName: string): Promise<NpmInfo | undefined> {
const raw = await this.fetchRawNpmInfo(escapedPackageName);
await sleep(0.01); // If we don't do this, npm resets the connection?
return raw === undefined ? undefined : npmInfoFromJson(raw);
}
async fetchRawNpmInfo(escapedPackageName: string): Promise<NpmInfoRaw | undefined> {
const info = await this.fetcher.fetchJson({
hostname: npmRegistryHostName,
path: escapedPackageName,
retries: true,
}) as { readonly error: string } | NpmInfoRaw;
if ("error" in info) {
if (info.error === "Not found") { return undefined; }
throw new Error(`Error getting version at ${escapedPackageName}: ${info.error}`);
}
if (!info["dist-tags"] && !info.versions) {
// Unpublished
return undefined;
}
return info;
}
// See https://github.com/npm/download-counts
async getDownloads(packageNames: ReadonlyArray<string>): Promise<ReadonlyArray<number>> {
// NPM uses a different API if there's only a single name, so ensure there's at least 2 for every batch of 128.
const names = (packageNames.length % 128) === 1 ? [...packageNames, "dummy"] : packageNames;
const nameGroups = Array.from(splitToFixedSizeGroups(names, 128)); // NPM has a limit of 128 packages at a time.
const out: number[] = [];
for (const nameGroup of nameGroups) {
const data = await this.fetcher.fetchJson({
hostname: npmApi,
path: `/downloads/point/last-month/${nameGroup.join(",")}`,
retries: true,
}) as { readonly error: string } | { readonly [key: string]: { readonly downloads: number } };
if ("error" in data) { throw new Error(data.error as string); }
for (const key of Object.keys(data)) {
assert(key === names[out.length], `at index ${out.length} of ${Object.keys(data).toString()} : ${key} !== ${names[out.length]}`);
out.push(data[key] ? data[key].downloads : 0);
}
}
return out;
}
}
function splitToFixedSizeGroups(names: ReadonlyArray<string>, chunkSize: number): ReadonlyArray<ReadonlyArray<string>> {
const out: string[][] = [];
for (let i = 0; i < names.length; i += chunkSize) {
out.push(names.slice(i, i + chunkSize));
}
return out;
}
export class NpmPublishClient {
static async create(config?: RegClient.Config, registry: Registry = Registry.NPM): Promise<NpmPublishClient> {
switch (registry) {
case Registry.NPM:
return new NpmPublishClient(new RegClient(config), { token: await getSecret(Secret.NPM_TOKEN) }, npmRegistry);
case Registry.Github:
return new NpmPublishClient(new RegClient(config), { token: await getSecret(Secret.GITHUB_PUBLISH_ACCESS_TOKEN) }, githubRegistry);
default:
assertNever(registry);
}
}
private constructor(private readonly client: RegClient, private readonly auth: RegClient.Credentials, private readonly registry: string) {}
async publish(publishedDirectory: string, packageJson: {}, dry: boolean, log: Logger): Promise<void> {
const readme = await readFile(joinPaths(publishedDirectory, "README.md"));
return new Promise<void>((resolve, reject) => {
const body = createTgz(publishedDirectory, reject);
const metadata = { readme, ...packageJson };
if (dry) {
log(`(dry) Skip publish of ${publishedDirectory} to ${this.registry}`);
}
resolve(dry ? undefined : promisifyVoid(cb => {
this.client.publish(this.registry, { access: "public", auth: this.auth, metadata, body }, cb);
}));
});
}
tag(packageName: string, version: string, distTag: string, dry: boolean, log: Logger): Promise<void> {
if (dry) {
log(`(dry) Skip tag of ${packageName}@${distTag} as ${version}`);
return Promise.resolve();
}
return promisifyVoid(cb => {
this.client.distTags.add(this.registry, { package: packageName, version, distTag, auth: this.auth }, cb);
});
}
deprecate(packageName: string, version: string, message: string): Promise<void> {
const url = resolveUrl(npmRegistry, packageName.replace("/", "%2f"));
const params = {
message,
version,
auth: this.auth,
};
return promisifyVoid(cb => { this.client.deprecate(url, params, cb); });
}
}
function npmInfoFromJson(n: NpmInfoRaw): NpmInfo {
return {
distTags: recordToMap(n["dist-tags"], identity),
// Callback ensures we remove any other properties
versions: recordToMap(n.versions, ({ typesPublisherContentHash, deprecated }) => ({ typesPublisherContentHash, deprecated })),
time: recordToMap(n.time),
};
}
function jsonFromNpmInfo(n: NpmInfo): NpmInfoRaw {
return {
"dist-tags": mapToRecord(n.distTags),
versions: mapToRecord(n.versions),
time: mapToRecord(n.time),
};
}
function promisifyVoid(callsBack: (cb: (error: Error | undefined) => void) => void): Promise<void> {
return new Promise<void>((resolve, reject) => {
callsBack(error => {
if (error) {
reject(error);
} else {
resolve();
}
});
});
}
================================================
FILE: src/lib/package-publisher.ts
================================================
import assert = require("assert");
import { TypeScriptVersion } from "definitelytyped-header-parser";
import { Logger } from "../util/logging";
import { joinPaths } from "../util/util";
import { readFileAndWarn, Registry } from "./common";
import { NpmPublishClient } from "./npm-client";
import { AnyPackage, NotNeededPackage } from "./packages";
import { ChangedTyping } from "./versions";
export async function publishTypingsPackage(
client: NpmPublishClient,
changedTyping: ChangedTyping,
dry: boolean,
log: Logger,
registry: Registry,
): Promise<void> {
const { pkg, version, latestVersion } = changedTyping;
await common(client, pkg, log, dry, registry);
if (pkg.isLatest) {
await updateTypeScriptVersionTags(pkg, version, client, log, dry);
}
assert((latestVersion === undefined) === pkg.isLatest);
if (latestVersion !== undefined) {
// If this is an older version of the package, we still update tags for the *latest*.
// NPM will update "latest" even if we are publishing an older version of a package (https://github.com/npm/npm/issues/6778),
// so we must undo that by re-tagging latest.
await updateLatestTag(pkg.fullNpmName, latestVersion, client, log, dry);
}
}
export async function publishNotNeededPackage(
client: NpmPublishClient,
pkg: NotNeededPackage,
dry: boolean,
log: Logger,
registry: Registry,
): Promise<void> {
log(`Deprecating ${pkg.name}`);
await common(client, pkg, log, dry, registry);
// Don't use a newline in the deprecation message because it will be displayed as "\n" and not as a newline.
await deprecateNotNeededPackage(client, pkg, dry, log);
}
async function common(client: NpmPublishClient, pkg: AnyPackage, log: Logger, dry: boolean, registry: Registry): Promise<void> {
const packageDir = pkg.outputDirectory;
const packageJson = await readFileAndWarn("generate", joinPaths(packageDir + (registry === Registry.Github ? "-github" : ""), "package.json"));
await client.publish(packageDir, packageJson, dry, log);
}
export async function deprecateNotNeededPackage(client: NpmPublishClient, pkg: NotNeededPackage, dry = false, log: Logger): Promise<void> {
const name = pkg.fullNpmName;
if (dry) {
log("(dry) Skip deprecate not needed package " + name + " at " + pkg.version.versionString);
} else {
log(`Deprecating ${name} at ${pkg.version.versionString} with message: ${pkg.deprecatedMessage()}.`);
await client.deprecate(name, pkg.version.versionString, pkg.deprecatedMessage());
}
}
export async function updateTypeScriptVersionTags(
pkg: AnyPackage, version: string, client: NpmPublishClient, log: Logger, dry: boolean,
): Promise<void> {
const tags = TypeScriptVersion.tagsToUpdate(pkg.minTypeScriptVersion);
const name = pkg.fullNpmName;
log(`Tag ${name}@${version} as ${JSON.stringify(tags)}`);
if (dry) {
log("(dry) Skip tag");
} else {
for (const tagName of tags) {
await client.tag(name, version, tagName, dry, log);
}
}
}
export async function updateLatestTag(
fullName: string, version: string, client: NpmPublishClient, log: Logger, dry: boolean): Promise<void> {
log(` but tag ${fullName}@${version} as "latest"`);
if (dry) {
log(" (dry) Skip move \"latest\" back to newest version");
} else {
await client.tag(fullName, version, "latest", dry, log);
}
}
================================================
FILE: src/lib/packages.test.ts
================================================
import { createMockDT } from "../mocks";
import { getTypingInfo } from "./definition-parser";
import { TypingsVersions } from "./packages";
describe(TypingsVersions, () => {
let versions: TypingsVersions;
beforeAll(() => {
const dt = createMockDT();
dt.addOldVersionOfPackage("jquery", "1");
dt.addOldVersionOfPackage("jquery", "2");
dt.addOldVersionOfPackage("jquery", "2.5");
versions = new TypingsVersions(getTypingInfo("jquery", dt.pkgFS("jquery")));
});
it("sorts the data from latest to oldest version", () => {
expect(Array.from(versions.getAll()).map(v => v.major)).toEqual([3, 2, 2, 1]);
});
it("returns the latest version", () => {
expect(versions.getLatest().major).toEqual(3);
});
it("finds the latest version when any version is wanted", () => {
expect(versions.get("*").major).toEqual(3);
});
it("finds the latest minor version for the given major version", () => {
expect(versions.get({ major: 2 }).major).toEqual(2);
expect(versions.get({ major: 2 }).minor).toEqual(5);
});
it("finds a specific version", () => {
expect(versions.get({ major: 2, minor: 0 }).major).toEqual(2);
expect(versions.get({ major: 2, minor: 0 }).minor).toEqual(0);
});
it("formats a version directory names", () => {
expect(versions.get({ major: 2, minor: 0 }).versionDirectoryName).toEqual("v2");
expect(versions.get({ major: 2, minor: 0 }).subDirectoryPath).toEqual("jquery/v2");
});
it("formats missing version error nicely", () => {
expect(() => versions.get({ major: 111, minor: 1001 })).toThrow("Could not find version 111.1001");
expect(() => versions.get({ major: 111 })).toThrow("Could not find version 111.*");
});
});
================================================
FILE: src/lib/packages.ts
================================================
import assert = require("assert");
import { AllTypeScriptVersion, Author, TypeScriptVersion } from "definitelytyped-header-parser";
import { FS } from "../get-definitely-typed";
import { assertSorted, joinPaths, mapValues, unmangleScopedPackage } from "../util/util";
import { readDataFile } from "./common";
import { outputDirPath, scopeName } from "./settings";
import { compare as compareSemver, Semver } from "./versions";
export class AllPackages {
static async read(dt: FS): Promise<AllPackages> {
return AllPackages.from(await readTypesDataFile(), readNotNeededPackages(dt));
}
static from(data: TypesDataFile, notNeeded: ReadonlyArray<NotNeededPackage>): AllPackages {
return new AllPackages(mapValues(new Map(Object.entries(data)), raw => new TypingsVersions(raw)), notNeeded);
}
static async readTypings(): Promise<ReadonlyArray<TypingsData>> {
return AllPackages.from(await readTypesDataFile(), []).allTypings();
}
static async readLatestTypings(): Promise<ReadonlyArray<TypingsData>> {
return AllPackages.from(await readTypesDataFile(), []).allLatestTypings();
}
/** Use for `--single` tasks only. Do *not* call this in a loop! */
static async readSingle(name: string): Promise<TypingsData> {
const data = await readTypesDataFile();
const raw = data[name];
if (!raw) {
throw new Error(`Can't find package ${name}`);
}
const versions = Object.keys(raw);
if (versions.length > 1) {
throw new Error(`Package ${name} has multiple versions.`);
}
return new TypingsData(raw[versions[0]], /*isLatest*/ true);
}
static readSingleNotNeeded(name: string, dt: FS): NotNeededPackage {
const notNeeded = readNotNeededPackages(dt);
const pkg = notNeeded.find(p => p.name === name);
if (pkg === undefined) {
throw new Error(`Cannot find not-needed package ${name}`);
}
return pkg;
}
private constructor(
private readonly data: ReadonlyMap<string, TypingsVersions>,
private readonly notNeeded: ReadonlyArray<NotNeededPackage>) {}
getNotNeededPackage(name: string): NotNeededPackage | undefined {
return this.notNeeded.find(p => p.name === name);
}
hasTypingFor(dep: PackageId): boolean {
return this.tryGetTypingsData(dep) !== undefined;
}
tryResolve(dep: PackageId): PackageId {
const versions = this.data.get(getMangledNameForScopedPackage(dep.name));
return versions ? versions.get(dep.version).id : dep;
}
/** Gets the latest version of a package. E.g. getLatest(node v6) was node v10 (before node v11 came out). */
getLatest(pkg: TypingsData): TypingsData {
return pkg.isLatest ? pkg : this.getLatestVersion(pkg.name);
}
private getLatestVersion(packageName: string): TypingsData {
const latest = this.tryGetLatestVersion(packageName);
if (!latest) {
throw new Error(`No such package ${packageName}.`);
}
return latest;
}
tryGetLatestVersion(packageName: string): TypingsData | undefined {
const versions = this.data.get(getMangledNameForScopedPackage(packageName));
return versions && versions.getLatest();
}
getTypingsData(id: PackageId): TypingsData {
const pkg = this.tryGetTypingsData(id);
if (!pkg) {
throw new Error(`No typings available for ${JSON.stringify(id)}`);
}
return pkg;
}
tryGetTypingsData({ name, version }: PackageId): TypingsData | undefined {
const versions = this.data.get(getMangledNameForScopedPackage(name));
return versions && versions.tryGet(version);
}
allPackages(): ReadonlyArray<AnyPackage> {
return [ ...this.allTypings(), ...this.allNotNeeded() ];
}
/** Note: this includes older version directories (`foo/v0`) */
allTypings(): ReadonlyArray<TypingsData> {
return assertSorted(Array.from(flattenData(this.data)), t => t.name);
}
allLatestTypings(): ReadonlyArray<TypingsData> {
return assertSorted(Array.from(this.data.values()).map(versions => versions.getLatest()), t => t.name);
}
allNotNeeded(): ReadonlyArray<NotNeededPackage> {
return this.notNeeded;
}
/** Returns all of the dependences *that have typings*, ignoring others, and including test dependencies. */
*allDependencyTypings(pkg: TypingsData): Iterable<TypingsData> {
for (const { name, version } of pkg.dependencies) {
const versions = this.data.get(getMangledNameForScopedPackage(name));
if (versions) {
yield versions.get(version);
}
}
for (const name of pkg.testDependencies) {
const versions = this.data.get(getMangledNameForScopedPackage(name));
if (versions) {
yield versions.getLatest();
}
}
}
}
// Same as the function in moduleNameResolver.ts in typescript
export function getMangledNameForScopedPackage(packageName: string): string {
if (packageName.startsWith("@")) {
const replaceSlash = packageName.replace("/", "__");
if (replaceSlash !== packageName) {
return replaceSlash.slice(1); // Take off the "@"
}
}
return packageName;
}
export const typesDataFilename = "definitions.json";
function* flattenData(data: ReadonlyMap<string, TypingsVersions>): Iterable<TypingsData> {
for (const versions of data.values()) {
yield* versions.getAll();
}
}
export type AnyPackage = NotNeededPackage | TypingsData;
interface BaseRaw {
/**
* The name of the library.
*
* A human readable version, e.g. it might be "Moment.js" even though `packageName` is "moment".
*/
readonly libraryName: string;
/**
* The NPM name to publish this under, e.g. "jquery".
*
* This does not include "@types".
*/
readonly typingsPackageName: string;
}
/** Prefer to use `AnyPackage` instead of this. */
export abstract class PackageBase {
static compare(a: PackageBase, b: PackageBase): number { return a.name.localeCompare(b.name); }
/** Note: for "foo__bar" this is still "foo__bar", not "@foo/bar". */
readonly name: string;
readonly libraryName: string;
get unescapedName(): string {
return unmangleScopedPackage(this.name) || this.name;
}
/** Short description for debug output. */
get desc(): string {
return this.isLatest ? this.name : `${this.name} v${this.major}.${this.minor}`;
}
constructor(data: BaseRaw) {
this.name = data.typingsPackageName;
this.libraryName = data.libraryName;
}
isNotNeeded(): this is NotNeededPackage {
return this instanceof NotNeededPackage;
}
abstract readonly isLatest: boolean;
abstract readonly projectName: string;
abstract readonly declaredModules: ReadonlyArray<string>;
abstract readonly globals: ReadonlyArray<string>;
abstract readonly minTypeScriptVersion: TypeScriptVersion;
/** '@types/foo' for a package 'foo'. */
get fullNpmName(): string {
return getFullNpmName(this.name);
}
/** '@types%2ffoo' for a package 'foo'. */
get fullEscapedNpmName(): string {
return `@${scopeName}%2f${this.name}`;
}
abstract readonly major: number;
abstract readonly minor: number;
get id(): PackageId {
return { name: this.name, version: { major: this.major, minor: this.minor }};
}
get outputDirectory(): string {
return joinPaths(outputDirPath, this.desc);
}
}
export function getFullNpmName(packageName: string): string {
return `@${scopeName}/${getMangledNameForScopedPackage(packageName)}`;
}
interface NotNeededPackageRaw extends BaseRaw {
/**
* If this is available, @types typings are deprecated as of this version.
* This is useful for packages that previously had DefinitelyTyped definitions but which now provide their own.
*/
// This must be "major.minor.patch"
readonly asOfVersion: string;
/** The package's own url, *not* DefinitelyTyped's. */
readonly sourceRepoURL: string;
}
export class NotNeededPackage extends PackageBase {
readonly version: Semver;
get license(): License.MIT { return License.MIT; }
readonly sourceRepoURL: string;
constructor(raw: NotNeededPackageRaw) {
super(raw);
this.sourceRepoURL = raw.sourceRepoURL;
for (const key of Object.keys(raw)) {
if (!["libraryName", "typingsPackageName", "sourceRepoURL", "asOfVersion"].includes(key)) {
throw new Error(`Unexpected key in not-needed package: ${key}`);
}
}
assert(raw.libraryName && raw.typingsPackageName && raw.sourceRepoURL && raw.asOfVersion);
this.version = Semver.parse(raw.asOfVersion);
}
get major(): number { return this.version.major; }
get minor(): number { return this.version.minor; }
// A not-needed package has no other versions. (TODO: allow that?)
get isLatest(): boolean { return true; }
get projectName(): string { return this.sourceRepoURL; }
get declaredModules(): ReadonlyArray<string> { return []; }
get globals(): ReadonlyArray<string> { return this.globals; }
get minTypeScriptVersion(): TypeScriptVersion { return TypeScriptVersion.lowest; }
readme(): string {
return `This is a stub types definition for ${this.libraryName} (${this.sourceRepoURL}).\n
${this.libraryName} provides its own type definitions, so you don't need ${getFullNpmName(this.name)} installed!`;
}
deprecatedMessage(): string {
return `This is a stub types definition. ${this.name} provides its own type definitions, so you do not need this installed.`;
}
}
export interface TypingsVersionsRaw {
[version: string]: TypingsDataRaw;
}
export interface TypingVersion {
major: number;
minor?: number;
}
export function formatTypingVersion(version: TypingVersion) {
return `${version.major}${version.minor === undefined ? "" : `.${version.minor}`}`;
}
/** If no version is specified, uses "*". */
export type DependencyVersion = TypingVersion | "*";
export function formatDependencyVersion(version: DependencyVersion) {
return version === "*" ? "*" : formatTypingVersion(version);
}
export interface PackageJsonDependency {
readonly name: string;
readonly version: string;
}
export interface TypingsDataRaw extends BaseRaw {
/**
* Other definitions, that exist in the same typings repo, that this package depends on.
*
* These will refer to *package names*, not *folder names*.
*/
readonly dependencies: ReadonlyArray<PackageId>;
/**
* Other definitions, that exist in the same typings repo, that the tests, but not the types, of this package depend on.
*
* These are always the latest version and will not include anything already in `dependencies`.
*/
readonly testDependencies: ReadonlyArray<string>;
/**
* External packages, from outside the typings repo, that provide definitions that this package depends on.
*/
readonly packageJsonDependencies: ReadonlyArray<PackageJsonDependency>;
/**
* Represents that there was a path mapping to a package.
*
* Not all path mappings are direct dependencies, they may be necessary for transitive dependencies. However, where `dependencies` and
* `pathMappings` share a key, they *must* share the same value.
*/
readonly pathMappings: ReadonlyArray<PathMapping>;
/**
* List of people that have contributed to the definitions in this package.
*
* These people will be requested for issue/PR review in the https://github.com/DefinitelyTyped/DefinitelyTyped repo.
*/
readonly contributors: ReadonlyArray<Author>;
/**
* The [older] version of the library that this definition package refers to, as represented *on-disk*.
*
* @note The latest version always exists in the root of the package tree and thus does not have a value for this property.
*/
readonly libraryVersionDirectoryName?: string;
/**
* Major version of the library.
*
* This data is parsed from a header comment in the entry point `.d.ts` and will be `0` if the file did not specify a version.
*/
readonly libraryMajorVersion: number;
/**
* Minor version of the library.
*
* This data is parsed from a header comment in the entry point `.d.ts` and will be `0` if the file did not specify a version.
*/
readonly libraryMinorVersion: number;
/**
* Minimum required TypeScript version to consume the definitions from this package.
*/
readonly minTsVersion: AllTypeScriptVersion;
/**
* List of TS versions that have their own directories, and corresponding "typesVersions" in package.json.
* Usually empty.
*/
readonly typesVersions: ReadonlyArray<TypeScriptVersion>;
/**
* Files that should be published with this definition, e.g. ["jquery.d.ts", "jquery-extras.d.ts"]
*
* Does *not* include a partial `package.json` because that will not be copied directly.
*/
readonly files: ReadonlyArray<string>;
/**
* The license that this definition package is released under.
*
* Can be either MIT or Apache v2, defaults to MIT when not explicitly defined in this package’s "package.json".
*/
readonly license: License;
/**
* A hash of the names and contents of the `files` list, used for versioning.
*/
readonly contentHash: string;
/**
* Name or URL of the project, e.g. "http://cordova.apache.org".
*/
readonly projectName: string;
/**
* A list of *values* declared in the global namespace.
*
* @note This does not include *types* declared in the global namespace.
*/
readonly globals: ReadonlyArray<string>;
/**
* External modules declared by this package. Includes the containing folder name when applicable (e.g. proper module).
*/
readonly declaredModules: ReadonlyArray<string>;
}
/**
* @see {TypingsDataRaw.pathMappings}
*/
export interface PathMapping {
readonly packageName: string;
readonly version: TypingVersion;
}
// TODO: support BSD -- but must choose a *particular* BSD license from the list at https://spdx.org/licenses/
export const enum License { MIT = "MIT", Apache20 = "Apache-2.0" }
const allLicenses = [License.MIT, License.Apache20];
export function getLicenseFromPackageJson(packageJsonLicense: unknown): License {
if (packageJsonLicense === undefined) { // tslint:disable-line strict-type-predicates (false positive)
return License.MIT;
}
if (typeof packageJsonLicense === "string" && packageJsonLicense === "MIT") {
throw new Error(`Specifying '"license": "MIT"' is redundant, this is the default.`);
}
if (allLicenses.includes(packageJsonLicense as License)) {
return packageJsonLicense as License;
}
throw new Error(`'package.json' license is ${JSON.stringify(packageJsonLicense)}.\nExpected one of: ${JSON.stringify(allLicenses)}}`);
}
export class TypingsVersions {
private readonly map: ReadonlyMap<Semver, TypingsData>;
/**
* Sorted from latest to oldest.
*/
private readonly versions: Semver[];
constructor(data: TypingsVersionsRaw) {
const versionMappings = new Map(Object.keys(data).map(key => {
const version = Semver.parse(key, true);
if (version) {
return [version, key];
}
throw new Error(`Unable to parse version ${key}`);
}));
/**
* Sorted from latest to oldest so that we publish the current version first.
* This is important because older versions repeatedly reset the "latest" tag to the current version.
*/
this.versions = Array.from(versionMappings.keys()).sort(compareSemver).reverse();
this.map = new Map(this.versions.map(version => {
const dataKey = versionMappings.get(version)!;
return [version, new TypingsData(data[dataKey], version.equals(this.versions[0]))];
}));
}
getAll(): Iterable<TypingsData> {
return this.map.values();
}
get(version: DependencyVersion): TypingsData {
return version === "*" ? this.getLatest() : this.getLatestMatch(version);
}
tryGet(version: DependencyVersion): TypingsData | undefined {
return version === "*" ? this.getLatest() : this.tryGetLatestMatch(version);
}
getLatest(): TypingsData {
return this.map.get(this.versions[0])!;
}
private getLatestMatch(version: TypingVersion): TypingsData {
const data = this.tryGetLatestMatch(version);
if (!data) {
throw new Error(`Could not find version ${version.major}.${version.minor ?? "*"}`);
}
return data;
}
private tryGetLatestMatch(version: TypingVersion): TypingsData | undefined {
const found = this.versions.find(v => v.major === version.major && (version.minor === undefined || v.minor === version.minor));
return found && this.map.get(found);
}
}
export class TypingsData extends PackageBase {
constructor(private readonly data: TypingsDataRaw, readonly isLatest: boolean) {
super(data);
}
get testDependencies(): ReadonlyArray<string> { return this.data.testDependencies; }
get contributors(): ReadonlyArray<Author> { return this.data.contributors; }
get major(): number { return this.data.libraryMajorVersion; }
get minor(): number { return this.data.libraryMinorVersion; }
get minTypeScriptVersion(): TypeScriptVersion {
return TypeScriptVersion.isSupported(this.data.minTsVersion) ? this.data.minTsVersion : TypeScriptVersion.lowest;
}
get typesVersions(): ReadonlyArray<TypeScriptVersion> { return this.data.typesVersions; }
get files(): ReadonlyArray<string> { return this.data.files; }
get license(): License { return this.data.license; }
get packageJsonDependencies(): ReadonlyArray<PackageJsonDependency> { return this.data.packageJsonDependencies; }
get contentHash(): string { return this.data.contentHash; }
get declaredModules(): ReadonlyArray<string> { return this.data.declaredModules; }
get projectName(): string { return this.data.projectName; }
get globals(): ReadonlyArray<string> { return this.data.globals; }
get pathMappings(): ReadonlyArray<PathMapping> { return this.data.pathMappings; }
get dependencies(): ReadonlyArray<PackageId> {
return this.data.dependencies;
}
get versionDirectoryName() {
return this.data.libraryVersionDirectoryName && `v${this.data.libraryVersionDirectoryName}`;
}
/** Path to this package, *relative* to the DefinitelyTyped directory. */
get subDirectoryPath(): string {
return this.isLatest ? this.name : `${this.name}/${this.versionDirectoryName}`;
}
}
/** Uniquely identifies a package. */
export interface PackageId {
readonly name: string;
readonly version: DependencyVersion;
}
export interface TypesDataFile {
readonly [packageName: string]: TypingsVersionsRaw;
}
function readTypesDataFile(): Promise<TypesDataFile> {
return readDataFile("parse-definitions", typesDataFilename) as Promise<TypesDataFile>;
}
export function readNotNeededPackages(dt: FS): ReadonlyArray<NotNeededPackage> {
const rawJson = dt.readJson("notNeededPackages.json"); // tslint:disable-line await-promise (tslint bug)
return (rawJson as { readonly packages: ReadonlyArray<NotNeededPackageRaw> }).packages.map(raw => new NotNeededPackage(raw));
}
================================================
FILE: src/lib/rolling-logs.ts
================================================
import BlobWriter, { readBlob } from "./azure-container";
export default class RollingLogs {
static async create(name: string, maxLines: number): Promise<RollingLogs> {
return new RollingLogs(name, maxLines, await BlobWriter.create());
}
private allLogs: string[] | undefined;
constructor(readonly name: string, readonly maxLines: number, private readonly container: BlobWriter) {}
async write(lines: string[]): Promise<void> {
const logs = this.allLogs || (this.allLogs = await this.readAllLogs());
const totalLines = logs.length + lines.length;
logs.splice(0, totalLines - this.maxLines);
logs.push(...lines);
await this.container.createBlobFromText(this.name, logs.join("\n"));
}
private async readAllLogs(): Promise<string[]> {
try {
return (await readBlob(this.name)).split("\n");
} catch (err) {
// 404
return [];
}
}
}
================================================
FILE: src/lib/secrets.ts
================================================
import { AuthenticationContext } from "adal-node";
import { KeyVaultClient, KeyVaultCredentials } from "azure-keyvault";
import { mapDefined } from "../util/util";
import { azureKeyvault } from "./settings";
export enum Secret {
/**
* Used to upload blobs.
* To find (or refresh) this value, go to https://ms.portal.azure.com -> All resources -> typespublisher -> General -> Access keys
*/
AZURE_STORAGE_ACCESS_KEY,
/**
* Lets the server update an issue (https://github.com/Microsoft/types-publisher/issues/40) on GitHub in case of an error.
* Create a token at: https://github.com/settings/tokens
*/
GITHUB_ACCESS_TOKEN,
/**
* This is used to ensure that only GitHub can send messages to our server.
* This should match the secret value set on GitHub: https://github.com/DefinitelyTyped/DefinitelyTyped/settings/hooks
* The Payload URL should be the URL of the Azure service.
* The webhook ignores the `sourceRepository` setting and can be triggered by *anything* with the secret,
* so make sure only DefinitelyTyped has the secret.
*/
GITHUB_SECRET,
/**
* Token used to perform request to NPM's API.
* This was generated by doing:
* - `npm login`
* - copy the token value (comes after `authToken=`) in `~/.npmrc`
* - `rm ~/.npmrc` (don't want to accidentally log out this token.)
*
* We only need one token in existence, so delete old tokens at: https://www.npmjs.com/settings/tokens
*/
NPM_TOKEN,
/**
* Token used to publish @definitelytyped/types-registry and @types/* to Github.
* This *could* be the same as GITHUB_ACCESS_TOKEN, but I think it's better if they remain separate.
*/
GITHUB_PUBLISH_ACCESS_TOKEN,
}
export const allSecrets: Secret[] = mapDefined(Object.keys(Secret), key => {
const value = (Secret as unknown as { [key: string]: unknown })[key];
return typeof value === "number" ? value : undefined; // tslint:disable-line strict-type-predicates (tslint bug)
});
export async function getSecret(secret: Secret): Promise<string> {
const clientId = process.env.TYPES_PUBLISHER_CLIENT_ID;
const clientSecret = process.env.TYPES_PUBLISHER_CLIENT_SECRET;
if (!(clientId && clientSecret)) {
throw new Error("Must set the TYPES_PUBLISHER_CLIENT_ID and TYPES_PUBLISHER_CLIENT_SECRET environment variables.");
}
// Copied from example usage at https://www.npmjs.com/package/azure-keyvault
const credentials = new KeyVaultCredentials((challenge, callback) => {
const context = new AuthenticationContext(challenge.authorization);
context.acquireTokenWithClientCredentials(challenge.resource, clientId, clientSecret, (error, tokenResponse) => {
if (error) {
throw error;
}
callback(undefined, `${tokenResponse!.tokenType} ${tokenResponse!.accessToken}`);
});
});
const client = new KeyVaultClient(credentials);
// Convert `AZURE_STORAGE_ACCESS_KEY` to `azure-storage-access-key` -- for some reason, Azure wouldn't allow secret names with underscores.
const azureSecretName = Secret[secret].toLowerCase().replace(/_/g, "-");
// console.log("Getting secret versions for: " + azureSecretName);
const versions = await client.getSecretVersions(azureKeyvault, azureSecretName);
versions.sort((a, b) => a.attributes.created.getTime() < b.attributes.created.getTime() ? 1 : -1);
// console.log(versions);
const urlParts = versions[0].id.split("/");
const latest = urlParts[urlParts.length - 1];
return (await client.getSecret(azureKeyvault, azureSecretName, latest)).value;
}
================================================
FILE: src/lib/settings.ts
================================================
import { readFileSync } from "fs";
import { join as joinPaths } from "path";
/** URL of the NPM registry to upload to. */
export const npmRegistryHostName = "registry.npmjs.org";
export const githubRegistryHostName = "npm.pkg.github.com";
export const npmRegistry = `https://${npmRegistryHostName}/`;
export const githubRegistry = `https://${githubRegistryHostName}/`;
export const npmApi = "api.npmjs.org";
/** Note: this is 'types' and not '@types' */
export const scopeName = "types";
const root = joinPaths(__dirname, "..", "..");
export const dataDirPath = joinPaths(root, "data");
export const outputDirPath = joinPaths(root, "output");
export const validateOutputPath = joinPaths(root, "validateOutput");
export const logDir = joinPaths(root, "logs");
/** URL to download the repository from. */
export const definitelyTypedZipUrl = "https://codeload.github.com/DefinitelyTyped/DefinitelyTyped/tar.gz/master";
/** The branch that DefinitelyTyped is sourced from. */
export const sourceBranch = "master";
/** Name of the azure storage account. Used for uploading data and logs. */
export const azureStorageAccount = "typespublisher";
/** Name of the azure container. */
export const azureContainer = "typespublisher";
/** URL of azure keyvault. */
export const azureKeyvault = "https://types-publisher-keys.vault.azure.net";
/** Issue in types-publisher that we will use to report webhook errors. */
export const errorsIssue = "Microsoft/types-publisher/issues/40";
export const typesDirectoryName = "types";
export const dependenciesWhitelist: ReadonlySet<string> =
new Set(readFileSync(joinPaths(root, "dependenciesWhitelist.txt"), "utf-8").split(/\r?\n/));
================================================
FILE: src/lib/versions.test.ts
================================================
import { Semver } from "./versions";
describe(Semver, () => {
it("returns a formatted description", () => {
expect(new Semver(1, 2, 3).versionString).toEqual("1.2.3");
});
it("parses semver versions", () => {
expect(Semver.parse("0.42.1").versionString).toEqual("0.42.1");
});
it("parses versions that do not strictly adhere to semver", () => {
expect(Semver.parse("1", true).versionString).toEqual("1.0.0");
expect(Semver.parse("0.42", true).versionString).toEqual("0.42.0");
});
it("throws when a version cannot be parsed", () => {
expect(() => Semver.parse("1")).toThrow();
expect(() => Semver.parse("1", false)).toThrow();
});
it("returns whether or not it's equal to another Semver", () => {
expect(Semver.parse("1.2.3").equals(new Semver(1, 2, 3))).toBe(true);
expect(Semver.parse("1.2.3").equals(new Semver(3, 2, 1))).toBe(false);
});
it("returns whether or not it's greater than another Semver", () => {
expect(Semver.parse("1.2.3").greaterThan(new Semver(1, 2, 2))).toBe(true);
expect(Semver.parse("1.2.3").equals(new Semver(1, 2, 4))).toBe(false);
});
});
================================================
FILE: src/lib/versions.ts
================================================
import { Logger } from "../util/logging";
import { assertDefined, best, intOfString } from "../util/util";
import { readDataFile } from "./common";
import { CachedNpmInfoClient } from "./npm-client";
import { AllPackages, NotNeededPackage, PackageId, TypingsData } from "./packages";
export const versionsFilename = "versions.json";
export interface ChangedTyping {
readonly pkg: TypingsData;
/** This is the version to be published, meaning it's the version that doesn't exist yet. */
readonly version: string;
/** For a non-latest version, this is the latest version; publishing an old version updates the 'latest' tag and we want to change it back. */
readonly latestVersion: string | undefined;
}
export interface ChangedPackagesJson {
readonly changedTypings: ReadonlyArray<ChangedTypingJson>;
readonly changedNotNeededPackages: ReadonlyArray<string>;
}
export interface ChangedTypingJson {
readonly id: PackageId;
readonly version: string;
readonly latestVersion?: string;
}
export interface ChangedPackages {
readonly changedTypings: ReadonlyArray<ChangedTyping>;
readonly changedNotNeededPackages: ReadonlyArray<NotNeededPackage>;
}
export async function readChangedPackages(allPackages: AllPackages): Promise<ChangedPackages> {
const json = await readDataFile("calculate-versions", versionsFilename) as ChangedPackagesJson;
return {
changedTypings: json.changedTypings.map(({ id, version, latestVersion }): ChangedTyping =>
({ pkg: allPackages.getTypingsData(id), version, latestVersion })),
changedNotNeededPackages: json.changedNotNeededPackages.map(id => assertDefined(allPackages.getNotNeededPackage(id))),
};
}
/**
* When we fail to publish a deprecated package, it leaves behind an entry in the time property.
* So the keys of 'time' give the actual 'latest'.
* If that's not equal to the expected latest, try again by bumping the patch version of the last attempt by 1.
*/
export function skipBadPublishes(pkg: NotNeededPackage, client: CachedNpmInfoClient, log: Logger) {
// because this is called right after isAlreadyDeprecated, we can rely on the cache being up-to-date
const info = assertDefined(client.getNpmInfoFromCache(pkg.fullEscapedNpmName));
const notNeeded = pkg.version;
const latest = Semver.parse(findActualLatest(info.time));
if (latest.equals(notNeeded) || latest.greaterThan(notNeeded) ||
info.versions.has(notNeeded.versionString) && !assertDefined(info.versions.get(notNeeded.versionString)).deprecated) {
const plusOne = new Semver(latest.major, latest.minor, latest.patch + 1);
log(`Deprecation of ${notNeeded.versionString} failed, instead using ${plusOne.versionString}.`);
return new NotNeededPackage({
asOfVersion: plusOne.versionString,
libraryName: pkg.libraryName,
sourceRepoURL: pkg.sourceRepoURL,
typingsPackageName: pkg.name,
});
}
return pkg;
}
function findActualLatest(times: Map<string, string>) {
const actual = best(
times, ([k, v], [bestK, bestV]) =>
(bestK === "modified" || bestK === "created") ? true :
(k === "modified" || k === "created") ? false :
new Date(v).getTime() > new Date(bestV).getTime());
if (!actual) {
throw new Error("failed to find actual latest");
}
return actual[0];
}
/** Version of a package published to NPM. */
export class Semver {
static parse(semver: string, coerce?: boolean): Semver {
const result = Semver.tryParse(semver, coerce);
if (!result) {
throw new Error(`Unexpected semver: ${semver}`);
}
return result;
}
static fromRaw({ major, minor, patch }: Semver): Semver {
return new Semver(major, minor, patch);
}
/**
* Per the semver spec <http://semver.org/#spec-item-2>:
*
* A normal version number MUST take the form X.Y.Z where X, Y, and Z are non-negative integers, and MUST NOT contain leading zeroes.
*
* @note This must parse the output of `versionString`.
*
* @param semver The version string.
* @param coerce Without this optional parameter the version MUST follow the above semver spec. However, when set to `true` components after the
* major version may be omitted. I.e. `1` equals `1.0` and `1.0.0`.
*/
static tryParse(semver: string, coerce?: boolean): Semver | undefined {
const rgx = /^(\d+)(\.(\d+))?(\.(\d+))?$/;
const match = rgx.exec(semver);
if (match) {
const { 1: major, 3: minor, 5: patch } = match;
if ((minor !== undefined && patch !== undefined) || coerce) { // tslint:disable-line:strict-type-predicates
return new Semver(intOfString(major), intOfString(minor || "0"), intOfString(patch || "0"));
}
}
return undefined;
}
constructor(readonly major: number, readonly minor: number, readonly patch: number) {}
get versionString(): string {
const { major, minor, patch } = this;
return `${major}.${minor}.${patch}`;
}
equals(other: Semver): boolean {
return compare(this, other) === 0;
}
greaterThan(other: Semver): boolean {
return compare(this, other) === 1;
}
}
/**
* Returns 0 if equal, 1 if x > y, -1 if x < y
*/
export function compare(x: Semver, y: Semver) {
const versions: Array<[number, number]> = [[x.major, y.major], [x.minor, y.minor], [x.patch, y.patch]];
for (const [componentX, componentY] of versions) {
if (componentX > componentY) {
return 1;
}
if (componentX < componentY) {
return -1;
}
}
return 0;
}
================================================
FILE: src/lib/webhook-server.ts
================================================
import { createHmac, timingSafeEqual } from "crypto";
import { createServer, IncomingMessage, Server, ServerResponse } from "http";
import { setInterval } from "timers";
import full from "../full";
import { Fetcher, stringOfStream } from "../util/io";
import { joinLogWithErrors, LoggerWithErrors, loggerWithErrors, LogWithErrors } from "../util/logging";
import { currentTimeStamp, parseJson } from "../util/util";
import { Options } from "./common";
import RollingLogs from "./rolling-logs";
import { sourceBranch } from "./settings";
export default async function webhookServer(
key: string,
githubAccessToken: string,
dry: boolean,
fetcher: Fetcher,
options: Options,
): Promise<Server> {
const fullOne = updateOneAtATime(async (log, timestamp) => {
log.info(""); log.info("");
log.info(`# ${timestamp}`);
log.info("");
log.info("Starting full...");
await full(dry, timestamp, githubAccessToken, fetcher, options, log);
});
setInterval((log, timestamp) => {
const result = fullOne(log, timestamp);
if (!result) { return; } // already working, so do nothing.
result.catch(e => { log.info(e.toString()); console.error(e); });
}, 2_000_000, loggerWithErrors()[0], currentTimeStamp());
return listenToGithub(key, fullOne);
}
function writeLog(rollingLogs: RollingLogs, logs: LogWithErrors): Promise<void> {
return rollingLogs.write(joinLogWithErrors(logs));
}
/** @param onUpdate: returns a promise in case it may error. Server will shut down on errors. */
function listenToGithub(
key: string,
onUpdate: (log: LoggerWithErrors, timeStamp: string) => Promise<void> | undefined,
): Server {
console.log("Before starting server");
const rollingLogs = RollingLogs.create("webhook-logs.md", 1000);
const server = createServer((req, resp) => {
switch (req.method) {
case "POST":
receiveUpdate(req, resp);
break;
default:
// Don't respond
}
});
return server;
function receiveUpdate(req: IncomingMessage, resp: ServerResponse): void {
const [log, logResult] = loggerWithErrors();
const timeStamp = currentTimeStamp();
try {
log.info("Before starting work");
work().then(() => rollingLogs.then(logs => writeLog(logs, logResult()))).catch(onError);
} catch (error) {
rollingLogs
.then(logs => writeLog(logs, logResult()))
.then(onError)
.catch(onError);
}
function onError(): void {
server.close();
}
async function work(): Promise<void> {
const data = await stringOfStream(req, "Request to webhook");
if (!checkSignature(key, data, req.headers, log)) {
return;
}
log.info(`Message from github: ${data.slice(0, 200)}...`);
const expectedRef = `refs/heads/${sourceBranch}`;
const actualRef = (parseJson(data) as { readonly ref: string }).ref;
if (actualRef === expectedRef) {
respond("Thanks for the update! Running full.");
await onUpdate(log, timeStamp);
} else {
const text = `Ignoring push to ${actualRef}, expected ${expectedRef}.`;
respond(text);
log.info(text);
}
}
// This is for the benefit of `npm run make-[production-]server-run`. GitHub ignores this.
function respond(text: string): void {
resp.write(text);
resp.end();
}
}
}
// Even if there are many changes to DefinitelyTyped in a row, we only perform one update at a time.
function updateOneAtATime(
doOnce: (log: LoggerWithErrors, timeStamp: string) => Promise<void>,
): (log: LoggerWithErrors, timeStamp: string) => Promise<void> | undefined {
let working = false;
return (log, timeStamp) => {
if (working) {
log.info("Not starting update, because already performing one.");
return undefined;
}
return work();
async function work(): Promise<void> {
log.info("Starting update");
working = true;
try {
await doOnce(log, timeStamp);
} catch (e) {
log.info(e.toString());
} finally {
working = false;
}
}
};
}
function checkSignature(key: string, data: string, headers: { readonly [key: string]: unknown }, log: LoggerWithErrors): boolean {
const signature = headers["x-hub-signature"];
const expected = expectedSignature(key, data);
// tslint:disable-next-line strict-type-predicates (TODO: tslint bug)
if (typeof signature === "string" && stringEqualsConstantTime(signature, expected)) {
return true;
}
// tslint:disable-next-line strict-string-expressions
log.error(`Invalid request: expected ${expected}, got ${signature}`);
log.error(`Headers are: ${JSON.stringify(headers, undefined, 4)}`);
log.error(`Data is: ${data}`);
log.error("");
return false;
}
// Use a constant-time compare to prevent timing attacks
function stringEqualsConstantTime(actual: string, expected: string): boolean {
// `timingSafeEqual` throws if they don't have the same length.
const actualBuffer = Buffer.alloc(expected.length);
actualBuffer.write(actual);
return timingSafeEqual(actualBuffer, Buffer.from(expected));
}
export function expectedSignature(key: string, data: string): string {
const hmac = createHmac("sha1", key);
hmac.write(data);
const digest = hmac.digest("hex");
return `sha1=${digest}`;
}
================================================
FILE: src/make-server-run.ts
================================================
import * as yargs from "yargs";
import { getSecret, Secret } from "./lib/secrets";
import { sourceBranch } from "./lib/settings";
import { expectedSignature } from "./lib/webhook-server";
import { makeHttpRequest } from "./util/io";
import { logUncaughtErrors } from "./util/util";
if (!module.parent) {
const remote = yargs.argv.remote;
logUncaughtErrors(main(remote ? { hostname: "typespublisher.azurewebsites.net" } : { hostname: "localhost", port: getPort() }));
}
function getPort(): number {
const port = parseInt(process.env.PORT!, 10);
if (!port) {
throw new Error("Must provide PORT");
}
return port;
}
async function main(options: { hostname: string, port?: number }): Promise<void> {
const key = await getSecret(Secret.GITHUB_SECRET);
const body = JSON.stringify({ ref: `refs/heads/${sourceBranch}` });
console.log(await makeHttpRequest({
hostname: options.hostname,
port: options.port,
path: "",
method: "POST",
body,
headers: { "x-hub-signature": expectedSignature(key, body) },
}));
}
================================================
FILE: src/mocks.ts
================================================
import { parseHeaderOrFail } from "definitelytyped-header-parser";
import { Dir, FS, InMemoryDT } from "./get-definitely-typed";
import { Semver } from "./lib/versions";
class DTMock {
public readonly fs: FS;
private readonly root: Dir;
constructor() {
this.root = new Dir(undefined);
this.root.set("notNeededPackages.json", `{
"packages": [{
"libraryName": "Angular 2",
"typingsPackageName": "angular",
"asOfVersion": "1.2.3",
"sourceRepoURL": "https://github.com/angular/angular2"
}]
}`);
this.fs = new InMemoryDT(this.root, "DefinitelyTyped");
}
public pkgDir(packageName: string): Dir {
return this.root.subdir("types").subdir(packageName);
}
public pkgFS(packageName: string): FS {
return this.fs.subDir("types").subDir(packageName);
}
/**
* Creates a shallow copy of a package, meaning all entries in the old version directory that will be created refer to the copied entry from the
* latest version. The only exceptions are the `index.d.ts` and `tsconfig.json` files.
*
* The directory name will exactly follow the given `olderVersion`. I.e. `2` will become `v2`, whereas `2.2` will become `v2.2`.
*
* @param packageName The package of which an old version is to be added.
* @param olderVersion The older version that's to be added.
*/
public addOldVersionOfPackage(packageName: string, olderVersion: string) {
const latestDir = this.pkgDir(packageName);
const index = latestDir.get("index.d.ts") as string;
const latestHeader = parseHeaderOrFail(index);
const latestVersion = `${latestHeader.libraryMajorVersion}.${latestHeader.libraryMinorVersion}`;
const olderVersionParsed = Semver.parse(olderVersion, true)!;
const oldDir = latestDir.subdir(`v${olderVersion}`);
const tsconfig = JSON.parse(latestDir.get("tsconfig.json") as string);
oldDir.set("index.d.ts", index.replace(latestVersion, `${olderVersionParsed.major}.${olderVersionParsed.minor}`));
oldDir.set("tsconfig.json", JSON.stringify({
...tsconfig,
compilerOptions: {
...tsconfig.compilerOptions,
paths: {
[packageName]: [`${packageName}/v${olderVersion}`],
},
},
}));
latestDir.forEach((content, entry) => {
if (
content !== oldDir
&& entry !== "index.d.ts"
&& entry !== "tsconfig.json"
&& !(content instanceof Dir && /^v\d+(\.\d+)?$/.test(entry))
) {
oldDir.set(entry, content);
}
});
return oldDir;
}
}
export function createMockDT() {
const dt = new DTMock();
const boring = dt.pkgDir("boring");
boring.set("index.d.ts", `// Type definitions for boring 1.0
// Project: https://boring.com
// Definitions by: Some Guy From Space <https://github.com/goodspaceguy420>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
import * as React from 'react';
export const drills: number;
`);
boring.set("secondary.d.ts", `
import deffo from 'react-default';
import { mammoths } from 'boring/quaternary';
export const hovercars: unknown;
declare module "boring/fake" {
import { stock } from 'boring/tertiary';
}
declare module "other" {
export const augmented: true;
}
`);
boring.set("tertiary.d.ts", `
import { stuff } from 'things';
export var stock: number;
`);
boring.set("quaternary.d.ts", `
export const mammoths: object;
`);
boring.set("commonjs.d.ts", `
import vortex = require('vorticon');
declare const australia: {};
export = australia;
`);
boring.set("v1.d.ts", `
export const inane: true | false;
`);
boring.set("untested.d.ts", `
import { help } from 'manual';
export const fungible: false;
`);
boring.set("boring-tests.ts", `
import { superstor } from "super-big-fun-hus";
import { drills } from "boring";
import { hovercars } from "boring/secondary";
import australia = require('boring/commonjs');
import { inane } from "boring/v1";
`);
boring.set("OTHER_FILES.txt", `
untested.d.ts
`);
boring.set("tsconfig.json", `{
"compilerOptions": {
"module": "commonjs",
"lib": [
"es6",
"dom"
],
"target": "es6",
"noImplicitAny": true,
"noImplicitThis": true,
"strictNullChecks": true,
"strictFunctionTypes": true,
"baseUrl": "../",
"typeRoots": [
"../"
],
"types": [],
"noEmit": true,
"forceConsistentCasingInFileNames": true
},
"files": [
"index.d.ts",
"boring-tests.ts"
]
}`);
const globby = dt.pkgDir("globby");
globby.set("index.d.ts", `// Type definitions for globby 0.2
// Project: https://globby-gloopy.com
// Definitions by: The Dragon Quest Slime <https://github.com/gloopyslime>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
/// <reference path="./sneaky.d.ts" />
/// <reference types="andere/snee" />
declare var x: number
`);
globby.set("merges.d.ts", `
declare var y: number
`);
globby.set("sneaky.d.ts", `
declare var ka: number
`);
globby.set("globby-tests.ts", `
var z = x;
`);
const tests = globby.subdir("test");
tests.set("other-tests.ts", `
/// <reference types="globby/merges" />
var z = y;
`);
globby.set("tsconfig.json", `{
"compilerOptions": {
"module": "commonjs",
"lib": [
"es6",
"dom"
],
"target": "es6",
"noImplicitAny": true,
"noImplicitThis": true,
"strictNullChecks": true,
"strictFunctionTypes": true,
"baseUrl": "../",
"typeRoots": [
"../"
],
"types": [],
"noEmit": true,
"forceConsistentCasingInFileNames": true
},
"files": [
"index.d.ts",
"globby-tests.ts",
"test/other-tests.ts"
]
}`);
const jquery = dt.pkgDir("jquery");
jquery.set("JQuery.d.ts", `
declare var jQuery: 1;
`);
jquery.set("index.d.ts", `// Type definitions for jquery 3.3
// Project: https://jquery.com
// Definitions by: Leonard Thieu <https://github.com/leonard-thieu>
// Definitions: https://github.com/DefinitelyTyped/DefinitelyTyped
// TypeScript Version: 2.3
/// <reference path="JQuery.d.ts" />
export = jQuery;
`);
jquery.set("jquery-tests.ts", `
console.log(jQuery);
`);
jquery.set("tsconfig.json", `{
"compilerOptions": {
"module": "commonjs",
"lib": [
"es6",
"dom"
],
"target": "es6",
"noImplicitAny": true,
"noImplicitThis": true,
"strictNullChecks": true,
"strictFunctionTypes": true,
"baseUrl": "../",
"typeRoots": [
"../"
],
"types": [],
"noEmit": true,
"forceConsistentCasingInFileNames": true
},
"files": [
"index.d.ts",
"jquery-tests.ts"
]
}
`);
return dt;
}
================================================
FILE: src/parse-definitions.test.ts
================================================
import { createMockDT } from "./mocks";
import parseDefinitions from "./parse-definitions";
import { loggerWithErrors } from "./util/logging";
import { testo } from "./util/test";
testo({
// async parseDefinitions() {
// const log = loggerWithErrors()[0]
// const dt = await getDefinitelyTyped(Options.defaults, log);
// const defs = await parseDefinitions(dt, undefined, log)
// expect(defs.allNotNeeded().length).toBeGreaterThan(0)
// expect(defs.allTypings().length).toBeGreaterThan(5000)
// const j = defs.tryGetLatestVersion("jquery")
// expect(j).toBeDefined()
// expect(j!.fullNpmName).toContain("types")
// expect(j!.fullNpmName).toContain("jquery")
// expect(defs.allPackages().length).toEqual(defs.allTypings().length + defs.allNotNeeded().length)
// },
async mockParse() {
const log = loggerWithErrors()[0];
const defs = await parseDefinitions(createMockDT().fs, undefined, log);
expect(defs.allNotNeeded().length).toBe(1);
expect(defs.allTypings().length).toBe(3);
const j = defs.tryGetLatestVersion("jquery");
expect(j).toBeDefined();
expect(j!.fullNpmName).toContain("types");
expect(j!.fullNpmName).toContain("jquery");
expect(defs.allPackages().length).toEqual(defs.allTypings().length + defs.allNotNeeded().length);
},
});
================================================
FILE: src/parse-definitions.ts
================================================
import * as yargs from "yargs";
import { FS, getDefinitelyTyped } from "./get-definitely-typed";
import { Options, writeDataFile } from "./lib/common";
import { getTypingInfo } from "./lib/definition-parser";
import { definitionParserWorkerFilename } from "./lib/definition-parser-worker";
import { AllPackages, readNotNeededPackages, typesDataFilename, TypingsVersionsRaw } from "./lib/packages";
import { parseNProcesses } from "./tester/test-runner";
import { LoggerWithErrors, loggerWithErrors } from "./util/logging";
import { assertDefined, filterNAtATimeOrdered, logUncaughtErrors, runWithChildProcesses } from "./util/util";
if (!module.parent) {
const singleName = yargs.argv.single as string | undefined;
const options = Options.defaults;
logUncaughtErrors(async () => {
const log = loggerWithErrors()[0];
const dt = await getDefinitelyTyped(options, log);
if (singleName) {
await single(singleName, dt);
} else {
await parseDefinitions(
dt,
options.parseInParallel
? { nProcesses: parseNProcesses(), definitelyTypedPath: assertDefined(options.definitelyTypedPath) }
: undefined,
log);
}
});
}
export interface ParallelOptions { readonly nProcesses: number; readonly definitelyTypedPath: string; }
export default async function parseDefinitions(dt: FS, parallel: ParallelOptions | undefined, log: LoggerWithErrors): Promise<AllPackages> {
log.info("Parsing definitions...");
const typesFS = dt.subDir("types");
const packageNames = await filterNAtATimeOrdered(parallel ? parallel.nProcesses : 1, typesFS.readdir(), name => typesFS.isDirectory(name));
log.info(`Found ${packageNames.length} packages.`);
const typings: { [name: string]: TypingsVersionsRaw } = {};
const start = Date.now();
if (parallel) {
log.info("Parsing in parallel...");
await runWithChildProcesses({
inputs: packageNames,
commandLineArgs: [`${parallel.definitelyTypedPath}/types`],
workerFile: definitionParserWorkerFilename,
nProcesses: parallel.nProcesses,
handleOutput({ data, packageName}: { data: TypingsVersionsRaw, packageName: string }) {
typings[packageName] = data;
},
});
} else {
log.info("Parsing non-parallel...");
for (const packageName of packageNames) {
typings[packageName] = getTypingInfo(packageName, typesFS.subDir(packageName));
}
}
log.info("Parsing took " + ((Date.now() - start) / 1000) + " s");
await writeDataFile(typesDataFilename, sorted(typings));
return AllPackages.from(typings, readNotNeededPackages(dt));
}
function sorted<T>(obj: { [name: string]: T }): { [name: string]: T } {
const out: { [name: string]: T } = {};
for (const key of Object.keys(obj).sort()) {
out[key] = obj[key];
}
return out;
}
async function single(singleName: string, dt: FS): Promise<void> {
const data = getTypingInfo(singleName, dt.subDir("types").subDir(singleName));
const typings = { [singleName]: data };
await writeDataFile(typesDataFilename, typings);
console.log(JSON.stringify(data, undefined, 4));
}
================================================
FILE: src/publish-packages.ts
================================================
import appInsights = require("applicationinsights");
import * as yargs from "yargs";
import { getDefinitelyTyped } from "./get-definitely-typed";
import { Options, Registry } from "./lib/common";
import { NpmPublishClient, UncachedNpmInfoClient, withNpmCache } from "./lib/npm-client";
import { deprecateNotNeededPackage, publishNotNeededPackage, publishTypingsPackage } from "./lib/package-publisher";
import { AllPackages } from "./lib/packages";
import { ChangedPackages, readChangedPackages, skipBadPublishes } from "./lib/versions";
import { Fetcher } from "./util/io";
import { logger, loggerWithErrors, writeLog } from "./util/logging";
import { logUncaughtErrors } from "./util/util";
if (!module.parent) {
const dry = !!yargs.argv.dry;
const deprecateName = yargs.argv.deprecate as string | undefined;
logUncaughtErrors(async () => {
const dt = await getDefinitelyTyped(Options.defaults, loggerWithErrors()[0]);
if (deprecateName !== undefined) {
// A '--deprecate' command is available in case types-publisher got stuck *while* trying to deprecate a package.
// Normally this should not be needed.
const log = logger()[0];
try {
await deprecateNotNeededPackage(
await NpmPublishClient.create(undefined, Registry.Github),
AllPackages.readSingleNotNeeded(deprecateName, dt),
false, /*dry*/
log,
);
} catch (e) {
// log and continue
log("publishing to github failed: " + e.toString());
}
await deprecateNotNeededPackage(
await NpmPublishClient.create(undefined, Registry.NPM), AllPackages.readSingleNotNeeded(deprecateName, dt), /*dry*/ false, log);
} else {
await publishPackages(await readChangedPackages(await AllPackages.read(dt)), dry, process.env.GH_API_TOKEN || "", new Fetcher());
}
});
}
export default async function publishPackages(
changedPackages: ChangedPackages,
dry: boolean,
githubAccessToken: string,
fetcher: Fetcher): Promise<void> {
const [log, logResult] =
gitextract_h4stda0m/ ├── .editorconfig ├── .gitignore ├── .travis.yml ├── .vscode/ │ └── launch.json ├── CODEOWNERS ├── DEBUGGING.md ├── IISNode.yml ├── LICENSE ├── README.md ├── dependenciesWhitelist.txt ├── jest.config.js ├── package.json ├── server.js ├── src/ │ ├── .vscode/ │ │ └── launch.json │ ├── calculate-versions.ts │ ├── check-parse-results.ts │ ├── clean.ts │ ├── crawl-npm.ts │ ├── create-search-index.ts │ ├── full.ts │ ├── generate-packages.test.ts │ ├── generate-packages.ts │ ├── get-definitely-typed.test.ts │ ├── get-definitely-typed.ts │ ├── index.ts │ ├── lib/ │ │ ├── azure-container.ts │ │ ├── blob-uploader.ts │ │ ├── common.ts │ │ ├── definition-parser-worker.ts │ │ ├── definition-parser.test.ts │ │ ├── definition-parser.ts │ │ ├── module-info.test.ts │ │ ├── module-info.ts │ │ ├── npm-client.ts │ │ ├── package-publisher.ts │ │ ├── packages.test.ts │ │ ├── packages.ts │ │ ├── rolling-logs.ts │ │ ├── secrets.ts │ │ ├── settings.ts │ │ ├── versions.test.ts │ │ ├── versions.ts │ │ └── webhook-server.ts │ ├── make-server-run.ts │ ├── mocks.ts │ ├── parse-definitions.test.ts │ ├── parse-definitions.ts │ ├── publish-packages.ts │ ├── publish-registry.ts │ ├── test-get-secrets.ts │ ├── tester/ │ │ ├── get-affected-packages.test.ts │ │ ├── get-affected-packages.ts │ │ ├── test-runner.test.ts │ │ ├── test-runner.ts │ │ └── test.ts │ ├── types/ │ │ ├── adal-node.d.ts │ │ ├── azure-keyvault.d.ts │ │ ├── fstream.d.ts │ │ └── npm-registry-client.d.ts │ ├── upload-blobs.ts │ ├── util/ │ │ ├── io.ts │ │ ├── logging.ts │ │ ├── progress.ts │ │ ├── test.ts │ │ ├── tgz.ts │ │ └── util.ts │ ├── validate.ts │ └── webhook.ts ├── tsconfig.json └── tslint.json
SYMBOL INDEX (502 symbols across 47 files)
FILE: src/calculate-versions.ts
function calculateVersions (line 15) | async function calculateVersions(
function computeAndSaveChangedPackages (line 28) | async function computeAndSaveChangedPackages(
function computeChangedPackages (line 42) | async function computeChangedPackages(
function fetchTypesPackageVersionInfo (line 78) | async function fetchTypesPackageVersionInfo(
function versionString (line 106) | function versionString(pkg: TypingsData, patch: number): string {
function isAlreadyDeprecated (line 110) | async function isAlreadyDeprecated(pkg: NotNeededPackage, client: Cached...
function getHighestVersionForMajor (line 123) | function getHighestVersionForMajor(versions: ReadonlyMap<string, NpmInfo...
function latestPatchMatchingMajorAndMinor (line 129) | function latestPatchMatchingMajorAndMinor(versions: Iterable<string>, ne...
function getLatestTypingVersion (line 141) | async function getLatestTypingVersion(pkg: TypingsData, client: CachedNp...
FILE: src/check-parse-results.ts
function checkParseResults (line 15) | async function checkParseResults(includeNpmChecks: boolean, dt: FS, opti...
function checkTypeScriptVersions (line 47) | function checkTypeScriptVersions(allPackages: AllPackages): void {
function checkPathMappings (line 57) | function checkPathMappings(allPackages: AllPackages): void {
function checkNpm (line 92) | async function checkNpm(
function packageHasTypes (line 137) | async function packageHasTypes(packageName: string, client: UncachedNpmI...
function getRegularVersions (line 142) | function getRegularVersions(versions: NpmInfoRawVersions): ReadonlyArray...
function versionHasTypes (line 149) | function versionHasTypes(info: NpmInfoVersion): boolean {
FILE: src/clean.ts
function clean (line 7) | function clean() {
FILE: src/crawl-npm.ts
function main (line 16) | async function main(options: Options): Promise<void> {
function allNpmPackages (line 30) | function allNpmPackages(): Promise<string[]> {
FILE: src/create-search-index.ts
type SearchRecord (line 21) | interface SearchRecord {
function createSearchIndex (line 36) | async function createSearchIndex(packages: AllPackages, client: Uncached...
function doSingle (line 43) | async function doSingle(name: string, client: UncachedNpmInfoClient): Pr...
function createSearchRecords (line 49) | async function createSearchRecords(packages: ReadonlyArray<TypingsData>,...
FILE: src/full.ts
function full (line 29) | async function full(
FILE: src/generate-packages.test.ts
function createRawPackage (line 7) | function createRawPackage(license: License): TypingsDataRaw {
function createTypesData (line 28) | function createTypesData(): TypesDataFile {
function createUnneededPackage (line 38) | function createUnneededPackage() {
method mitLicenseText (line 47) | mitLicenseText() {
method apacheLicenseText (line 51) | apacheLicenseText() {
method basicReadme (line 55) | basicReadme() {
method readmeContainsProjectName (line 59) | readmeContainsProjectName() {
method readmeOneDependency (line 63) | readmeOneDependency() {
method readmeNoGlobals (line 67) | readmeNoGlobals() {
method basicPackageJson (line 71) | basicPackageJson() {
method githubPackageJsonName (line 102) | githubPackageJsonName() {
method githubPackageJsonRegistry (line 108) | githubPackageJsonRegistry() {
method basicNotNeededPackageJson (line 115) | basicNotNeededPackageJson() {
method scopedNotNeededPackageJson (line 132) | scopedNotNeededPackageJson() {
method githubNotNeededPackageJson (line 155) | githubNotNeededPackageJson() {
FILE: src/generate-packages.ts
function generatePackages (line 31) | async function generatePackages(dt: FS, allPackages: AllPackages, change...
function generateTypingPackage (line 53) | async function generateTypingPackage(typing: TypingsData, packages: AllP...
function generateNotNeededPackage (line 65) | async function generateNotNeededPackage(pkg: NotNeededPackage, client: C...
function writeCommonOutputs (line 71) | async function writeCommonOutputs(pkg: AnyPackage, packageJson: string, ...
function outputFilePath (line 85) | async function outputFilePath(pkg: AnyPackage, registry: Registry, filen...
type Dependencies (line 94) | interface Dependencies { [name: string]: string; }
function createPackageJSON (line 96) | function createPackageJSON(typing: TypingsData, version: string, package...
function getDependencies (line 132) | function getDependencies(packageJsonDependencies: ReadonlyArray<PackageJ...
function dependencySemver (line 148) | function dependencySemver(dependency: DependencyVersion): string {
function createNotNeededPackageJSON (line 152) | function createNotNeededPackageJSON(
function createReadme (line 184) | function createReadme(typing: TypingsData): string {
function getLicenseFileText (line 217) | function getLicenseFileText(typing: AnyPackage): string {
function apacheLicense (line 228) | function apacheLicense(typing: TypingsData): string {
FILE: src/get-definitely-typed.test.ts
method downloadDefinitelyTyped (line 7) | async downloadDefinitelyTyped() {
method createDirs (line 12) | createDirs() {
method simpleMemoryFS (line 18) | simpleMemoryFS() {
FILE: src/get-definitely-typed.ts
type FS (line 19) | interface FS {
function getDefinitelyTyped (line 49) | async function getDefinitelyTyped(options: Options, log: LoggerWithError...
function getLocallyInstalledDefinitelyTyped (line 63) | function getLocallyInstalledDefinitelyTyped(path: string): FS {
function downloadAndExtractFile (line 67) | function downloadAndExtractFile(url: string): Promise<FS> {
type ReadonlyDir (line 109) | interface ReadonlyDir extends ReadonlyMap<string, ReadonlyDir | string> {
class Dir (line 114) | class Dir extends Map<string, Dir | string> implements ReadonlyDir {
method constructor (line 115) | constructor(readonly parent: Dir | undefined) { super(); }
method subdir (line 117) | subdir(name: string): Dir {
method finish (line 130) | finish(): Dir {
class InMemoryDT (line 140) | class InMemoryDT implements FS {
method constructor (line 142) | constructor(readonly curDir: ReadonlyDir, readonly pathToRoot: string) {}
method tryGetEntry (line 144) | private tryGetEntry(path: string): ReadonlyDir | string | undefined {
method getEntry (line 165) | private getEntry(path: string): ReadonlyDir | string {
method getDir (line 171) | private getDir(dirPath: string): Dir {
method readFile (line 179) | readFile(filePath: string): string {
method readdir (line 187) | readdir(dirPath?: string): ReadonlyArray<string> {
method readJson (line 191) | readJson(path: string): unknown {
method isDirectory (line 195) | isDirectory(path: string): boolean {
method exists (line 199) | exists(path: string): boolean {
method subDir (line 203) | subDir(path: string): FS {
method debugPath (line 207) | debugPath(): string {
class DiskFS (line 212) | class DiskFS implements FS {
method constructor (line 213) | constructor(private readonly rootPrefix: string) {
method getPath (line 217) | private getPath(path: string | undefined): string {
method readdir (line 225) | readdir(dirPath?: string): ReadonlyArray<string> {
method isDirectory (line 229) | isDirectory(dirPath: string): boolean {
method readJson (line 233) | readJson(path: string): unknown {
method readFile (line 237) | readFile(path: string): string {
method exists (line 241) | exists(path: string): boolean {
method subDir (line 245) | subDir(path: string): FS {
method debugPath (line 249) | debugPath(): string {
function validatePath (line 255) | function validatePath(path: string): void {
FILE: src/lib/azure-container.ts
class BlobWriter (line 12) | class BlobWriter {
method create (line 13) | static async create(): Promise<BlobWriter> {
method constructor (line 17) | private constructor(private readonly service: BlobService) {}
method setCorsProperties (line 19) | setCorsProperties(): Promise<void> {
method ensureCreated (line 36) | ensureCreated(options: BlobService.CreateContainerOptions): Promise<vo...
method createBlobFromFile (line 42) | createBlobFromFile(blobName: string, fileName: string): Promise<void> {
method createBlobFromText (line 46) | createBlobFromText(blobName: string, text: string): Promise<void> {
method listBlobs (line 50) | async listBlobs(prefix: string): Promise<BlobService.BlobResult[]> {
method deleteBlob (line 67) | deleteBlob(blobName: string): Promise<void> {
method createBlobFromStream (line 73) | private createBlobFromStream(blobName: string, stream: NodeJS.Readable...
function readBlob (line 85) | async function readBlob(blobName: string): Promise<string> {
function readJsonBlob (line 105) | async function readJsonBlob(blobName: string): Promise<object> {
function urlOfBlob (line 109) | function urlOfBlob(blobName: string): string {
function promisifyErrorOrResult (line 113) | function promisifyErrorOrResult<A>(callsBack: (x: ErrorOrResult<A>) => v...
function promisifyErrorOrResponse (line 125) | function promisifyErrorOrResponse(callsBack: (x: ErrorOrResponse) => voi...
FILE: src/lib/blob-uploader.ts
function uploadBlobsAndUpdateIssue (line 12) | async function uploadBlobsAndUpdateIssue(timeStamp: string): Promise<voi...
function uploadBlobs (line 21) | async function uploadBlobs(container: BlobWriter, timeStamp: string): Pr...
function logsUploadedLocation (line 39) | function logsUploadedLocation(timeStamp: string): string {
function uploadLogs (line 43) | async function uploadLogs(container: BlobWriter, timeStamp: string, log:...
function uploadDirectory (line 48) | async function uploadDirectory(
function logAndUploadFile (line 63) | async function logAndUploadFile(container: BlobWriter, blobName: string,...
function uploadFile (line 69) | async function uploadFile(container: BlobWriter, blobName: string, fileP...
function deleteDirectory (line 75) | async function deleteDirectory(container: BlobWriter, uploadedDirPath: s...
function removeOldDirectories (line 82) | async function removeOldDirectories(container: BlobWriter, prefix: strin...
function uploadIndex (line 105) | function uploadIndex(container: BlobWriter, timeStamp: string, dataUrls:...
FILE: src/lib/common.ts
type Registry (line 15) | enum Registry {
type Options (line 23) | interface Options {
type TesterOptions (line 40) | interface TesterOptions extends Options {
function readDataFile (line 45) | function readDataFile(generatedBy: string, fileName: string): Promise<ob...
function readFileAndWarn (line 50) | async function readFileAndWarn(generatedBy: string, filePath: string): P...
function writeDataFile (line 59) | async function writeDataFile(filename: string, content: {}, formatted = ...
function dataFilePath (line 64) | function dataFilePath(filename: string): string {
FILE: src/lib/definition-parser.ts
function matchesVersion (line 22) | function matchesVersion(typingsDataRaw: TypingsDataRaw, version: TypingV...
function formattedLibraryVersion (line 29) | function formattedLibraryVersion(typingsDataRaw: TypingsDataRaw) {
function getTypingInfo (line 34) | function getTypingInfo(packageName: string, fs: FS): TypingsVersionsRaw {
type LsMinusTypesVersionsAndPackageJson (line 92) | interface LsMinusTypesVersionsAndPackageJson {
function getTypesVersionsAndPackageJson (line 98) | function getTypesVersionsAndPackageJson(ls: ReadonlyArray<string>): LsMi...
function parseVersionFromDirectoryName (line 123) | function parseVersionFromDirectoryName(directoryName: string): TypingVer...
function combineDataForAllTypesVersions (line 134) | function combineDataForAllTypesVersions(
function getAllUniqueValues (line 183) | function getAllUniqueValues<K extends string, T>(records: ReadonlyArray<...
type TypingDataFromIndividualTypeScriptVersion (line 187) | interface TypingDataFromIndividualTypeScriptVersion {
function getTypingDataForSingleTypesVersion (line 205) | function getTypingDataForSingleTypesVersion(
function checkPackageJsonDependencies (line 249) | function checkPackageJsonDependencies(dependencies: unknown, path: strin...
function checkFilesFromTsConfig (line 286) | function checkFilesFromTsConfig(packageName: string, tsconfig: TsConfig,...
type TsConfig (line 318) | interface TsConfig {
type DependenciesAndPathMappings (line 325) | interface DependenciesAndPathMappings { readonly dependencies: ReadonlyA...
function calculateDependencies (line 326) | function calculateDependencies(
function parseDependencyVersionFromPath (line 406) | function parseDependencyVersionFromPath(packageName: string, dependencyN...
function withoutEnd (line 415) | function withoutEnd(s: string, end: string): string | undefined {
function hash (line 422) | function hash(files: ReadonlyArray<string>, tsconfigPathsForHash: Readon...
function readFileAndThrowOnBOM (line 431) | function readFileAndThrowOnBOM(fileName: string, fs: FS): string {
function checkAllFilesUsed (line 446) | function checkAllFilesUsed(ls: ReadonlyArray<string>, usedFiles: Set<str...
function checkAllUsedRecur (line 456) | function checkAllUsedRecur(ls: Iterable<string>, usedFiles: Set<string>,...
FILE: src/lib/module-info.test.ts
function getBoringReferences (line 9) | function getBoringReferences() {
method allReferencedFilesFromTsconfigFiles (line 13) | allReferencedFilesFromTsconfigFiles() {
method allReferencedFilesFromTestIncludesSecondaryInternalFiles (line 18) | allReferencedFilesFromTestIncludesSecondaryInternalFiles() {
method allReferencedFilesFromTsconfigGlobal (line 23) | allReferencedFilesFromTsconfigGlobal() {
method allReferencedFilesFromTestIncludesSecondaryTripleSlashTypes (line 28) | allReferencedFilesFromTestIncludesSecondaryTripleSlashTypes() {
method getModuleInfoWorksWithOtherFiles (line 38) | getModuleInfoWorksWithOtherFiles() {
method getModuleInfoForNestedTypeReferences (line 48) | getModuleInfoForNestedTypeReferences() {
method versionTypeRefThrows (line 59) | versionTypeRefThrows() {
method selfVersionTypeRefAllowed (line 73) | selfVersionTypeRefAllowed() {
method getTestDependenciesWorks (line 88) | getTestDependenciesWorks() {
FILE: src/lib/module-info.ts
function getModuleInfo (line 10) | function getModuleInfo(packageName: string, all: Map<string, ts.SourceFi...
function sourceFileExportsSomething (line 88) | function sourceFileExportsSomething({ statements }: ts.SourceFile): bool...
type ModuleInfo (line 102) | interface ModuleInfo {
function properModuleName (line 114) | function properModuleName(folderName: string, fileName: string): string {
function rootName (line 125) | function rootName(importText: string, typeFiles: Map<string, unknown>, p...
function withoutExtension (line 141) | function withoutExtension(str: string, ext: string): string {
function allReferencedFiles (line 147) | function allReferencedFiles(
function resolveModule (line 183) | function resolveModule(importSpecifier: string, fs: FS): string {
type Reference (line 200) | interface Reference {
function findReferencedFiles (line 211) | function findReferencedFiles(src: ts.SourceFile, packageName: string, su...
function parseRequire (line 301) | function parseRequire(reference: ts.ExternalModuleReference): string {
function isValueNamespace (line 309) | function isValueNamespace(ns: ts.ModuleDeclaration): boolean {
function statementDeclaresValue (line 318) | function statementDeclaresValue(statement: ts.Statement): boolean {
function assertNoWindowsSlashes (line 339) | function assertNoWindowsSlashes(packageName: string, fileName: string): ...
function getTestDependencies (line 346) | function getTestDependencies(
function createSourceFile (line 407) | function createSourceFile(filename: string, content: string): ts.SourceF...
FILE: src/lib/npm-client.ts
type NpmInfoCache (line 17) | type NpmInfoCache = ReadonlyMap<string, NpmInfo>;
type NpmInfoRaw (line 19) | interface NpmInfoRaw {
type NpmInfoRawVersions (line 28) | interface NpmInfoRawVersions {
type NpmInfo (line 33) | interface NpmInfo {
type NpmInfoVersion (line 38) | interface NpmInfoVersion {
type CachedNpmInfoClient (line 43) | interface CachedNpmInfoClient {
function withNpmCache (line 48) | async function withNpmCache<T>(uncachedClient: UncachedNpmInfoClient, cb...
class UncachedNpmInfoClient (line 83) | class UncachedNpmInfoClient {
method fetchNpmInfo (line 86) | async fetchNpmInfo(escapedPackageName: string): Promise<NpmInfo | unde...
method fetchRawNpmInfo (line 92) | async fetchRawNpmInfo(escapedPackageName: string): Promise<NpmInfoRaw ...
method getDownloads (line 110) | async getDownloads(packageNames: ReadonlyArray<string>): Promise<Reado...
function splitToFixedSizeGroups (line 132) | function splitToFixedSizeGroups(names: ReadonlyArray<string>, chunkSize:...
class NpmPublishClient (line 140) | class NpmPublishClient {
method create (line 141) | static async create(config?: RegClient.Config, registry: Registry = Re...
method constructor (line 152) | private constructor(private readonly client: RegClient, private readon...
method publish (line 154) | async publish(publishedDirectory: string, packageJson: {}, dry: boolea...
method tag (line 169) | tag(packageName: string, version: string, distTag: string, dry: boolea...
method deprecate (line 179) | deprecate(packageName: string, version: string, message: string): Prom...
function npmInfoFromJson (line 190) | function npmInfoFromJson(n: NpmInfoRaw): NpmInfo {
function jsonFromNpmInfo (line 199) | function jsonFromNpmInfo(n: NpmInfo): NpmInfoRaw {
function promisifyVoid (line 207) | function promisifyVoid(callsBack: (cb: (error: Error | undefined) => voi...
FILE: src/lib/package-publisher.ts
function publishTypingsPackage (line 12) | async function publishTypingsPackage(
function publishNotNeededPackage (line 33) | async function publishNotNeededPackage(
function common (line 46) | async function common(client: NpmPublishClient, pkg: AnyPackage, log: Lo...
function deprecateNotNeededPackage (line 52) | async function deprecateNotNeededPackage(client: NpmPublishClient, pkg: ...
function updateTypeScriptVersionTags (line 62) | async function updateTypeScriptVersionTags(
function updateLatestTag (line 77) | async function updateLatestTag(
FILE: src/lib/packages.ts
class AllPackages (line 11) | class AllPackages {
method read (line 12) | static async read(dt: FS): Promise<AllPackages> {
method from (line 16) | static from(data: TypesDataFile, notNeeded: ReadonlyArray<NotNeededPac...
method readTypings (line 20) | static async readTypings(): Promise<ReadonlyArray<TypingsData>> {
method readLatestTypings (line 23) | static async readLatestTypings(): Promise<ReadonlyArray<TypingsData>> {
method readSingle (line 28) | static async readSingle(name: string): Promise<TypingsData> {
method readSingleNotNeeded (line 41) | static readSingleNotNeeded(name: string, dt: FS): NotNeededPackage {
method constructor (line 50) | private constructor(
method getNotNeededPackage (line 54) | getNotNeededPackage(name: string): NotNeededPackage | undefined {
method hasTypingFor (line 58) | hasTypingFor(dep: PackageId): boolean {
method tryResolve (line 62) | tryResolve(dep: PackageId): PackageId {
method getLatest (line 68) | getLatest(pkg: TypingsData): TypingsData {
method getLatestVersion (line 72) | private getLatestVersion(packageName: string): TypingsData {
method tryGetLatestVersion (line 80) | tryGetLatestVersion(packageName: string): TypingsData | undefined {
method getTypingsData (line 85) | getTypingsData(id: PackageId): TypingsData {
method tryGetTypingsData (line 93) | tryGetTypingsData({ name, version }: PackageId): TypingsData | undefin...
method allPackages (line 98) | allPackages(): ReadonlyArray<AnyPackage> {
method allTypings (line 103) | allTypings(): ReadonlyArray<TypingsData> {
method allLatestTypings (line 107) | allLatestTypings(): ReadonlyArray<TypingsData> {
method allNotNeeded (line 111) | allNotNeeded(): ReadonlyArray<NotNeededPackage> {
method allDependencyTypings (line 116) | *allDependencyTypings(pkg: TypingsData): Iterable<TypingsData> {
function getMangledNameForScopedPackage (line 134) | function getMangledNameForScopedPackage(packageName: string): string {
type AnyPackage (line 152) | type AnyPackage = NotNeededPackage | TypingsData;
type BaseRaw (line 154) | interface BaseRaw {
method compare (line 172) | static compare(a: PackageBase, b: PackageBase): number { return a.name.l...
method unescapedName (line 178) | get unescapedName(): string {
method desc (line 183) | get desc(): string {
method constructor (line 187) | constructor(data: BaseRaw) {
method isNotNeeded (line 192) | isNotNeeded(): this is NotNeededPackage {
method fullNpmName (line 203) | get fullNpmName(): string {
method fullEscapedNpmName (line 208) | get fullEscapedNpmName(): string {
method id (line 215) | get id(): PackageId {
method outputDirectory (line 219) | get outputDirectory(): string {
function getFullNpmName (line 224) | function getFullNpmName(packageName: string): string {
type NotNeededPackageRaw (line 228) | interface NotNeededPackageRaw extends BaseRaw {
class NotNeededPackage (line 239) | class NotNeededPackage extends PackageBase {
method license (line 242) | get license(): License.MIT { return License.MIT; }
method constructor (line 246) | constructor(raw: NotNeededPackageRaw) {
method major (line 260) | get major(): number { return this.version.major; }
method minor (line 261) | get minor(): number { return this.version.minor; }
method isLatest (line 264) | get isLatest(): boolean { return true; }
method projectName (line 265) | get projectName(): string { return this.sourceRepoURL; }
method declaredModules (line 266) | get declaredModules(): ReadonlyArray<string> { return []; }
method globals (line 267) | get globals(): ReadonlyArray<string> { return this.globals; }
method minTypeScriptVersion (line 268) | get minTypeScriptVersion(): TypeScriptVersion { return TypeScriptVersi...
method readme (line 270) | readme(): string {
method deprecatedMessage (line 275) | deprecatedMessage(): string {
type TypingsVersionsRaw (line 280) | interface TypingsVersionsRaw {
type TypingVersion (line 284) | interface TypingVersion {
function formatTypingVersion (line 289) | function formatTypingVersion(version: TypingVersion) {
type DependencyVersion (line 294) | type DependencyVersion = TypingVersion | "*";
function formatDependencyVersion (line 296) | function formatDependencyVersion(version: DependencyVersion) {
type PackageJsonDependency (line 300) | interface PackageJsonDependency {
type TypingsDataRaw (line 305) | interface TypingsDataRaw extends BaseRaw {
type PathMapping (line 412) | interface PathMapping {
type License (line 418) | const enum License { MIT = "MIT", Apache20 = "Apache-2.0" }
function getLicenseFromPackageJson (line 420) | function getLicenseFromPackageJson(packageJsonLicense: unknown): License {
class TypingsVersions (line 433) | class TypingsVersions {
method constructor (line 441) | constructor(data: TypingsVersionsRaw) {
method getAll (line 462) | getAll(): Iterable<TypingsData> {
method get (line 466) | get(version: DependencyVersion): TypingsData {
method tryGet (line 470) | tryGet(version: DependencyVersion): TypingsData | undefined {
method getLatest (line 474) | getLatest(): TypingsData {
method getLatestMatch (line 478) | private getLatestMatch(version: TypingVersion): TypingsData {
method tryGetLatestMatch (line 486) | private tryGetLatestMatch(version: TypingVersion): TypingsData | undef...
class TypingsData (line 492) | class TypingsData extends PackageBase {
method constructor (line 493) | constructor(private readonly data: TypingsDataRaw, readonly isLatest: ...
method testDependencies (line 497) | get testDependencies(): ReadonlyArray<string> { return this.data.testD...
method contributors (line 498) | get contributors(): ReadonlyArray<Author> { return this.data.contribut...
method major (line 499) | get major(): number { return this.data.libraryMajorVersion; }
method minor (line 500) | get minor(): number { return this.data.libraryMinorVersion; }
method minTypeScriptVersion (line 502) | get minTypeScriptVersion(): TypeScriptVersion {
method typesVersions (line 505) | get typesVersions(): ReadonlyArray<TypeScriptVersion> { return this.da...
method files (line 507) | get files(): ReadonlyArray<string> { return this.data.files; }
method license (line 508) | get license(): License { return this.data.license; }
method packageJsonDependencies (line 509) | get packageJsonDependencies(): ReadonlyArray<PackageJsonDependency> { ...
method contentHash (line 510) | get contentHash(): string { return this.data.contentHash; }
method declaredModules (line 511) | get declaredModules(): ReadonlyArray<string> { return this.data.declar...
method projectName (line 512) | get projectName(): string { return this.data.projectName; }
method globals (line 513) | get globals(): ReadonlyArray<string> { return this.data.globals; }
method pathMappings (line 514) | get pathMappings(): ReadonlyArray<PathMapping> { return this.data.path...
method dependencies (line 516) | get dependencies(): ReadonlyArray<PackageId> {
method versionDirectoryName (line 520) | get versionDirectoryName() {
method subDirectoryPath (line 525) | get subDirectoryPath(): string {
type PackageId (line 531) | interface PackageId {
type TypesDataFile (line 536) | interface TypesDataFile {
function readTypesDataFile (line 539) | function readTypesDataFile(): Promise<TypesDataFile> {
function readNotNeededPackages (line 543) | function readNotNeededPackages(dt: FS): ReadonlyArray<NotNeededPackage> {
FILE: src/lib/rolling-logs.ts
class RollingLogs (line 3) | class RollingLogs {
method create (line 4) | static async create(name: string, maxLines: number): Promise<RollingLo...
method constructor (line 10) | constructor(readonly name: string, readonly maxLines: number, private ...
method write (line 12) | async write(lines: string[]): Promise<void> {
method readAllLogs (line 20) | private async readAllLogs(): Promise<string[]> {
FILE: src/lib/secrets.ts
type Secret (line 8) | enum Secret {
function getSecret (line 49) | async function getSecret(secret: Secret): Promise<string> {
FILE: src/lib/versions.ts
type ChangedTyping (line 10) | interface ChangedTyping {
type ChangedPackagesJson (line 18) | interface ChangedPackagesJson {
type ChangedTypingJson (line 23) | interface ChangedTypingJson {
type ChangedPackages (line 29) | interface ChangedPackages {
function readChangedPackages (line 34) | async function readChangedPackages(allPackages: AllPackages): Promise<Ch...
function skipBadPublishes (line 48) | function skipBadPublishes(pkg: NotNeededPackage, client: CachedNpmInfoCl...
function findActualLatest (line 67) | function findActualLatest(times: Map<string, string>) {
class Semver (line 80) | class Semver {
method parse (line 81) | static parse(semver: string, coerce?: boolean): Semver {
method fromRaw (line 89) | static fromRaw({ major, minor, patch }: Semver): Semver {
method tryParse (line 104) | static tryParse(semver: string, coerce?: boolean): Semver | undefined {
method constructor (line 116) | constructor(readonly major: number, readonly minor: number, readonly p...
method versionString (line 118) | get versionString(): string {
method equals (line 123) | equals(other: Semver): boolean {
method greaterThan (line 127) | greaterThan(other: Semver): boolean {
function compare (line 135) | function compare(x: Semver, y: Semver) {
FILE: src/lib/webhook-server.ts
function webhookServer (line 14) | async function webhookServer(
function writeLog (line 36) | function writeLog(rollingLogs: RollingLogs, logs: LogWithErrors): Promis...
function listenToGithub (line 41) | function listenToGithub(
function updateOneAtATime (line 105) | function updateOneAtATime(
function checkSignature (line 131) | function checkSignature(key: string, data: string, headers: { readonly [...
function stringEqualsConstantTime (line 147) | function stringEqualsConstantTime(actual: string, expected: string): boo...
function expectedSignature (line 154) | function expectedSignature(key: string, data: string): string {
FILE: src/make-server-run.ts
function getPort (line 14) | function getPort(): number {
function main (line 22) | async function main(options: { hostname: string, port?: number }): Promi...
FILE: src/mocks.ts
class DTMock (line 6) | class DTMock {
method constructor (line 10) | constructor() {
method pkgDir (line 23) | public pkgDir(packageName: string): Dir {
method pkgFS (line 27) | public pkgFS(packageName: string): FS {
method addOldVersionOfPackage (line 40) | public addOldVersionOfPackage(packageName: string, olderVersion: strin...
function createMockDT (line 76) | function createMockDT() {
FILE: src/parse-definitions.test.ts
method mockParse (line 19) | async mockParse() {
FILE: src/parse-definitions.ts
type ParallelOptions (line 31) | interface ParallelOptions { readonly nProcesses: number; readonly defini...
function parseDefinitions (line 32) | async function parseDefinitions(dt: FS, parallel: ParallelOptions | unde...
function sorted (line 63) | function sorted<T>(obj: { [name: string]: T }): { [name: string]: T } {
function single (line 71) | async function single(singleName: string, dt: FS): Promise<void> {
FILE: src/publish-packages.ts
function publishPackages (line 43) | async function publishPackages(
function postGithub (line 157) | async function postGithub(path: string, data: any, githubToken: string, ...
function queryGithub (line 176) | async function queryGithub(path: string, githubToken: string, fetcher: F...
FILE: src/publish-registry.ts
function publishRegistry (line 29) | async function publishRegistry(dt: FS, allPackages: AllPackages, dry: bo...
function isSevenDaysAfter (line 81) | function isSevenDaysAfter(time: Date): boolean {
function generate (line 87) | async function generate(registry: string, packageJson: {}): Promise<void> {
function publish (line 106) | async function publish(client: NpmPublishClient, packageName: string, pa...
function installForValidate (line 120) | async function installForValidate(log: Logger): Promise<void> {
function validate (line 142) | async function validate(log: Logger): Promise<void> {
function validateIsSubset (line 150) | async function validateIsSubset(notNeeded: ReadonlyArray<NotNeededPackag...
function assertJsonNewer (line 162) | function assertJsonNewer(newer: { [s: string]: any }, older: { [s: strin...
function generatePackageJson (line 189) | function generatePackageJson(name: string, registryName: RegistryName, v...
type Registry (line 217) | interface Registry {
function generateRegistry (line 224) | async function generateRegistry(typings: ReadonlyArray<TypingsData>, cli...
type ProcessedNpmInfo (line 250) | interface ProcessedNpmInfo {
function fetchAndProcessNpmInfo (line 257) | async function fetchAndProcessNpmInfo(escapedPackageName: string, client...
function getLatestVersion (line 266) | function getLatestVersion(versions: Iterable<string>): Semver {
FILE: src/test-get-secrets.ts
function main (line 10) | async function main(): Promise<void> {
FILE: src/tester/get-affected-packages.test.ts
method updatedPackage (line 20) | updatedPackage() {
method deletedPackage (line 26) | deletedPackage() {
FILE: src/tester/get-affected-packages.ts
type Affected (line 4) | interface Affected {
function getAffectedPackages (line 11) | function getAffectedPackages(allPackages: AllPackages, changedPackageIds...
function allDependencies (line 20) | function allDependencies(allPackages: AllPackages, packages: Iterable<Ty...
function collectDependers (line 25) | function collectDependers(changedPackages: PackageId[], reverseDependenc...
function sortPackages (line 34) | function sortPackages(packages: Iterable<TypingsData>): TypingsData[] {
function transitiveClosure (line 38) | function transitiveClosure<T>(initialItems: Iterable<T>, getRelatedItems...
function getReverseDependencies (line 64) | function getReverseDependencies(allPackages: AllPackages, changedPackage...
function packageIdToKey (line 92) | function packageIdToKey(pkg: PackageId): string {
FILE: src/tester/test-runner.test.ts
method ok (line 28) | ok() {
method forgotToDeleteFiles (line 31) | forgotToDeleteFiles() {
method tooManyDeletes (line 38) | tooManyDeletes() {
method extraneousFile (line 42) | extraneousFile() {
method forgotToUpdateNotNeededJson (line 50) | forgotToUpdateNotNeededJson() {
method scoped (line 54) | scoped() {
method missingSource (line 77) | missingSource() {
method missingTypings (line 81) | missingTypings() {
method missingTypingsLatest (line 85) | missingTypingsLatest() {
method deprecatedSameVersion (line 89) | deprecatedSameVersion() {
method deprecatedOlderVersion (line 99) | deprecatedOlderVersion() {
method missingNpmVersion (line 109) | missingNpmVersion() {
method olderNpmVersion (line 118) | olderNpmVersion() {
method ok (line 125) | ok() {
FILE: src/tester/test-runner.ts
type GitDiff (line 46) | interface GitDiff {
function testAffectedOnly (line 51) | async function testAffectedOnly(options: TesterOptions): Promise<void> {
function parseNProcesses (line 58) | function parseNProcesses(): number {
function testerOptions (line 70) | function testerOptions(runFromDefinitelyTyped: boolean): TesterOptions {
function runTests (line 76) | async function runTests(
function getAffectedPackagesFromDiff (line 92) | async function getAffectedPackagesFromDiff(dt: FS, definitelyTypedPath: ...
function getNotNeededPackages (line 118) | function getNotNeededPackages(allPackages: AllPackages, diffs: GitDiff[]...
function checkNotNeededPackage (line 140) | function checkNotNeededPackage(unneeded: NotNeededPackage, source: NpmIn...
function doInstalls (line 159) | async function doInstalls(allPackages: AllPackages, packages: Iterable<T...
function directoryPath (line 183) | function directoryPath(typesPath: string, pkg: TypingsData): string {
function doRunTests (line 187) | async function doRunTests(
type TesterError (line 275) | interface TesterError {
function runCommand (line 279) | async function runCommand(log: LoggerWithErrors, cwd: string | undefined...
function gitChanges (line 298) | function gitChanges(diffs: GitDiff[]): PackageId[] {
function gitDiff (line 328) | async function gitDiff(log: Logger, definitelyTypedPath: string): Promis...
function getDependencyFromFile (line 361) | function getDependencyFromFile(file: string): PackageId | undefined {
FILE: src/tester/test.ts
function main (line 20) | async function main(options: TesterOptions, nProcesses: number, all: boo...
FILE: src/types/adal-node.d.ts
class AuthenticationContext (line 1) | class AuthenticationContext {
type TokenResponse (line 9) | interface TokenResponse {
FILE: src/types/azure-keyvault.d.ts
type AuthenticatorCallback (line 1) | type AuthenticatorCallback = (error: Error | null | undefined, authoriza...
class KeyVaultCredentials (line 2) | class KeyVaultCredentials {
type Challenge (line 5) | interface Challenge {
class KeyVaultClient (line 10) | class KeyVaultClient {
type SecretVersion (line 16) | interface SecretVersion {
type SecretBundle (line 25) | interface SecretBundle {
FILE: src/types/fstream.d.ts
type ReaderOptions (line 2) | interface ReaderOptions {
type FStreamEntry (line 7) | interface FStreamEntry {
FILE: src/types/npm-registry-client.d.ts
class RegClient (line 2) | class RegClient {
type Config (line 13) | interface Config {
type RequestParams (line 16) | interface RequestParams {
type PublishParams (line 20) | interface PublishParams {
type AddTagParams (line 26) | interface AddTagParams {
type DeprecateParams (line 32) | interface DeprecateParams {
type Credentials (line 37) | interface Credentials {
FILE: src/util/io.ts
function readFile (line 15) | async function readFile(path: string): Promise<string> {
function readFileSync (line 23) | function readFileSync(path: string): string {
function readJsonSync (line 31) | function readJsonSync(path: string): object {
function readJson (line 35) | async function readJson(path: string): Promise<object> {
function writeFile (line 39) | function writeFile(path: string, content: string): Promise<void> {
function writeJson (line 43) | function writeJson(path: string, content: unknown, formatted = true): Pr...
function streamOfString (line 47) | function streamOfString(text: string): NodeJS.ReadableStream {
function stringOfStream (line 54) | function stringOfStream(stream: NodeJS.ReadableStream, description: stri...
function streamDone (line 73) | function streamDone(stream: NodeJS.WritableStream): Promise<void> {
type FetchOptions (line 79) | interface FetchOptions {
class Fetcher (line 88) | class Fetcher {
method fetchJson (line 91) | async fetchJson(options: FetchOptions): Promise<unknown> {
method fetch (line 100) | async fetch(options: FetchOptions): Promise<string> {
function makeHttpRequest (line 117) | function makeHttpRequest(options: FetchOptions): Promise<string> {
function doRequest (line 121) | function doRequest(options: FetchOptions, makeRequest: typeof request, a...
function sleep (line 145) | async function sleep(seconds: number): Promise<void> {
function isDirectory (line 149) | async function isDirectory(path: string): Promise<boolean> {
FILE: src/util/logging.ts
type Logger (line 9) | type Logger = (message: string) => void;
type Log (line 12) | type Log = string[];
type LoggerWithErrors (line 15) | interface LoggerWithErrors {
type LogWithErrors (line 21) | interface LogWithErrors {
function quietLogger (line 33) | function quietLogger(): [Logger, () => Log] {
function alsoConsoleLogger (line 39) | function alsoConsoleLogger(consoleLog: Logger): [Logger, () => Log] {
function logger (line 51) | function logger(): [Logger, () => Log] {
function loggerWithErrorsHelper (line 56) | function loggerWithErrorsHelper(loggerOrQuietLogger: () => [Logger, () =...
function quietLoggerWithErrors (line 66) | function quietLoggerWithErrors(): [LoggerWithErrors, () => LogWithErrors] {
function loggerWithErrors (line 71) | function loggerWithErrors(): [LoggerWithErrors, () => LogWithErrors] {
function moveLogs (line 79) | function moveLogs(dest: Logger, src: Log, mapper?: (message: string) => ...
function moveLogsWithErrors (line 86) | function moveLogsWithErrors(dest: LoggerWithErrors, {infos, errors}: Log...
function logPath (line 91) | function logPath(logName: string): string {
function writeLog (line 95) | async function writeLog(logName: string, contents: ReadonlyArray<string>...
function joinLogWithErrors (line 100) | function joinLogWithErrors({infos, errors}: LogWithErrors): Log {
FILE: src/util/progress.ts
type Options (line 3) | interface Options {
class ProgressBar (line 12) | class ProgressBar {
method constructor (line 23) | constructor(options: Options) {
method update (line 29) | update(current: number, flavor?: string): void {
method doUpdate (line 41) | private doUpdate(current: number): void {
method done (line 57) | done(): void {
class UpdatableConsole (line 65) | class UpdatableConsole {
method update (line 68) | update(action: (charm: charm.CharmInstance) => void): void {
method end (line 75) | end(): void {
function strProgress (line 87) | function strProgress(str: string): number {
FILE: src/util/test.ts
function testo (line 3) | function testo(o: { [s: string]: () => void }) {
function createTypingsVersionRaw (line 9) | function createTypingsVersionRaw(
FILE: src/util/tgz.ts
function gzip (line 8) | function gzip(input: NodeJS.ReadableStream): NodeJS.ReadableStream {
function unGzip (line 12) | function unGzip(input: NodeJS.ReadableStream): NodeJS.ReadableStream {
function writeTgz (line 18) | function writeTgz(inputDirectory: string, outFileName: string): Promise<...
function createTgz (line 26) | function createTgz(dir: string, onError: (error: Error) => void): NodeJS...
function createTar (line 30) | function createTar(dir: string, onError: (error: Error) => void): NodeJS...
function addDirectoryExecutablePermission (line 43) | function addDirectoryExecutablePermission(entry: FStreamEntry): boolean {
function addExecutePermissionsFromReadPermissions (line 50) | function addExecutePermissionsFromReadPermissions(mode: number): number {
FILE: src/util/util.ts
function assertDefined (line 11) | function assertDefined<T>(x: T | undefined, message?: string | Error | u...
constant DEFAULT_CRASH_RECOVERY_MAX_OLD_SPACE_SIZE (line 16) | const DEFAULT_CRASH_RECOVERY_MAX_OLD_SPACE_SIZE = 4096;
function parseJson (line 18) | function parseJson(text: string): object {
function currentTimeStamp (line 26) | function currentTimeStamp(): string {
type ProgressOptions (line 33) | interface ProgressOptions<T, U> {
function nAtATime (line 39) | async function nAtATime<T, U>(
function filter (line 68) | function filter<T>(iterable: Iterable<T>, predicate: (value: T) => boole...
type Awaitable (line 83) | type Awaitable<T> = T | Promise<T>;
function filterNAtATimeOrdered (line 85) | async function filterNAtATimeOrdered<T>(
function unique (line 91) | function unique<T>(arr: Iterable<T>): T[] {
function logUncaughtErrors (line 95) | function logUncaughtErrors(promise: Promise<unknown> | (() => Promise<un...
function initArray (line 102) | function initArray<T>(length: number, makeElement: (i: number) => T): T[] {
function joinPaths (line 111) | function joinPaths(...paths: string[]): string {
function normalizeSlashes (line 116) | function normalizeSlashes(path: string): string {
function hasWindowsSlashes (line 120) | function hasWindowsSlashes(path: string): boolean {
function intOfString (line 124) | function intOfString(str: string): number {
function sortObjectKeys (line 132) | function sortObjectKeys<T extends { [key: string]: unknown }>(data: T): T {
function exec (line 141) | function exec(cmd: string, cwd?: string): Promise<{ error: Error | undef...
function execAndThrowErrors (line 154) | async function execAndThrowErrors(cmd: string, cwd?: string): Promise<st...
function best (line 166) | function best<T>(inputs: Iterable<T>, isBetter: (a: T, b: T) => boolean)...
function computeHash (line 187) | function computeHash(content: string): string {
function mapValues (line 196) | function mapValues<K, V1, V2>(map: Map<K, V1>, valueMapper: (value: V1) ...
function mapDefined (line 204) | function mapDefined<T, U>(arr: Iterable<T>, mapper: (t: T) => U | undefi...
function mapDefinedAsync (line 215) | async function mapDefinedAsync<T, U>(arr: Iterable<T>, mapper: (t: T) =>...
function sort (line 238) | function sort<T>(values: Iterable<T>, comparer?: (a: T, b: T) => number)...
function join (line 242) | function join<T>(values: Iterable<T>, joiner = ", "): string {
type RunWithChildProcessesOptions (line 251) | interface RunWithChildProcessesOptions<In> {
function runWithChildProcesses (line 258) | function runWithChildProcesses<In>(
type CrashRecoveryState (line 310) | const enum CrashRecoveryState {
type RunWithListeningChildProcessesOptions (line 317) | interface RunWithListeningChildProcessesOptions<In> {
function runWithListeningChildProcesses (line 330) | function runWithListeningChildProcesses<In>(
type MaxOldSpaceSizeArgument (line 548) | interface MaxOldSpaceSizeArgument {
function getMaxOldSpaceSizeArg (line 554) | function getMaxOldSpaceSizeArg(argv: ReadonlyArray<string>): MaxOldSpace...
function getMaxOldSpaceSize (line 568) | function getMaxOldSpaceSize(argv: ReadonlyArray<string>): number | undef...
function getExecArgvWithoutMaxOldSpaceSize (line 575) | function getExecArgvWithoutMaxOldSpaceSize(): ReadonlyArray<string> {
function assertNever (line 589) | function assertNever(_: never): never {
function recordToMap (line 595) | function recordToMap<T, U>(record: Record<string, T>, cb?: (t: T) => U):...
function mapToRecord (line 605) | function mapToRecord<T, U>(map: Map<string, T>, cb?: (t: T) => U): Recor...
function identity (line 611) | function identity<T>(t: T): T { return t; }
function withoutStart (line 613) | function withoutStart(s: string, start: string): string | undefined {
function unmangleScopedPackage (line 618) | function unmangleScopedPackage(packageName: string): string | undefined {
function split (line 624) | function split<T, U>(inputs: ReadonlyArray<T>, cb: (t: T) => U | undefin...
function assertSorted (line 636) | function assertSorted<T>(a: ReadonlyArray<T>, cb: (t: T) => string = (t:...
FILE: src/validate.ts
function validate (line 32) | async function validate(dt: FS): Promise<void> {
function doAll (line 36) | async function doAll(): Promise<void> {
function doValidate (line 42) | async function doValidate(packageNames: ReadonlyArray<string>): Promise<...
function validatePackages (line 52) | async function validatePackages(packageNames: ReadonlyArray<string>, out...
function validatePackage (line 89) | async function validatePackage(packageName: string, outputDirecory: stri...
function writePackage (line 116) | async function writePackage(packageDirectory: string, packageName: strin...
function runCommand (line 145) | async function runCommand(commandDescription: string, log: LoggerWithErr...
FILE: src/webhook.ts
function main (line 14) | async function main(): Promise<void> {
Condensed preview — 70 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (307K chars).
[
{
"path": ".editorconfig",
"chars": 147,
"preview": "root = true\n\n[*]\ncharset = utf-8\nend_of_line = lf\nindent_size = 4\nindent_style = space\ninsert_final_newline = true\ntrim_"
},
{
"path": ".gitignore",
"chars": 71,
"preview": "bin/\ncache/\ndata/\nlogs/\nnode_modules/\noutput/\nvalidateOutput/\n\\#*\n.\\#*\n"
},
{
"path": ".travis.yml",
"chars": 120,
"preview": "language: node_js\n\nnode_js:\n - 'node'\n\nsudo: false\n\nbranches:\n only:\n - master\n\ninstall:\n - npm ci\n\ngit:\n depth: 1\n"
},
{
"path": ".vscode/launch.json",
"chars": 1146,
"preview": "{\n // Use IntelliSense to learn about possible attributes.\n // Hover to view descriptions of existing attributes.\n"
},
{
"path": "CODEOWNERS",
"chars": 26,
"preview": "* @sandersn @RyanCavanaugh"
},
{
"path": "DEBUGGING.md",
"chars": 1483,
"preview": "# DEBUGGING\n\nDebugging an Azure App Service has been a miserable learning\nexperience for me, so here are some tips to he"
},
{
"path": "IISNode.yml",
"chars": 286,
"preview": "# For all options see https://github.com/tjanczuk/iisnode/blob/master/src/samples/configuration/iisnode.yml\nnodeProcessC"
},
{
"path": "LICENSE",
"chars": 1162,
"preview": " MIT License\n\n Copyright (c) Microsoft Corporation. All rights reserved.\n\n Permission is hereby granted, free o"
},
{
"path": "README.md",
"chars": 435,
"preview": "# This repo has moved.\n\nThis service is now part of [microsoft/DefinitelyTyped-tools](https://github.com/microsoft/Defin"
},
{
"path": "dependenciesWhitelist.txt",
"chars": 2742,
"preview": "@babel/code-frame\n@babel/core\n@babel/generator\n@babel/parser\n@babel/template\n@babel/traverse\n@babel/types\n@elastic/elast"
},
{
"path": "jest.config.js",
"chars": 267,
"preview": "module.exports = {\n \"roots\": [\n \"src\"\n ],\n \"preset\": \"ts-jest\",\n \"testRegex\": \"(/__tests__/.*|\\\\.(tes"
},
{
"path": "package.json",
"chars": 3633,
"preview": "{\n \"private\": true,\n \"name\": \"types-publisher\",\n \"version\": \"0.0.0\",\n \"main\": \"./bin/index.js\",\n \"types\": \"./bin/in"
},
{
"path": "server.js",
"chars": 60,
"preview": "require(\"./bin/webhook.js\").default().catch(console.error);\n"
},
{
"path": "src/.vscode/launch.json",
"chars": 470,
"preview": "{\n // Use IntelliSense to learn about possible attributes.\n // Hover to view descriptions of existing attributes.\n"
},
{
"path": "src/calculate-versions.ts",
"chars": 7177,
"preview": "import assert = require(\"assert\");\n\nimport { FS, getDefinitelyTyped } from \"./get-definitely-typed\";\nimport { Options, w"
},
{
"path": "src/check-parse-results.ts",
"chars": 8615,
"preview": "import { FS, getDefinitelyTyped } from \"./get-definitely-typed\";\nimport { Options } from \"./lib/common\";\nimport { NpmInf"
},
{
"path": "src/clean.ts",
"chars": 226,
"preview": "import { removeSync } from \"fs-extra\";\n\nif (!module.parent) {\n clean();\n}\n\nexport function clean() {\n for (const d"
},
{
"path": "src/crawl-npm.ts",
"chars": 2052,
"preview": "import assert = require(\"assert\");\nimport oboe = require(\"oboe\");\n\nimport { packageHasTypes } from \"./check-parse-result"
},
{
"path": "src/create-search-index.ts",
"chars": 2371,
"preview": "import * as yargs from \"yargs\";\n\nimport { getDefinitelyTyped } from \"./get-definitely-typed\";\nimport { Options, writeDat"
},
{
"path": "src/full.ts",
"chars": 2201,
"preview": "import appInsights = require(\"applicationinsights\");\nimport * as yargs from \"yargs\";\n\nimport calculateVersions from \"./c"
},
{
"path": "src/generate-packages.test.ts",
"chars": 6426,
"preview": "import { createNotNeededPackageJSON, createPackageJSON, createReadme, getLicenseFileText } from \"./generate-packages\";\ni"
},
{
"path": "src/generate-packages.ts",
"chars": 10682,
"preview": "import { makeTypesVersionsForPackageJson } from \"definitelytyped-header-parser\";\nimport { emptyDir, mkdir, mkdirp, readF"
},
{
"path": "src/get-definitely-typed.test.ts",
"chars": 1050,
"preview": "import { Dir, FS, getDefinitelyTyped, InMemoryDT } from \"./get-definitely-typed\";\nimport { Options } from \"./lib/common\""
},
{
"path": "src/get-definitely-typed.ts",
"chars": 9361,
"preview": "import appInsights = require(\"applicationinsights\");\nimport assert = require(\"assert\");\nimport { ensureDir, pathExistsSy"
},
{
"path": "src/index.ts",
"chars": 652,
"preview": "export { getDefinitelyTyped } from \"./get-definitely-typed\";\nexport { withNpmCache, CachedNpmInfoClient, NpmPublishClien"
},
{
"path": "src/lib/azure-container.ts",
"chars": 4994,
"preview": "import { BlobService, common, createBlobService, ErrorOrResponse, ErrorOrResult } from \"azure-storage\";\nimport * as fs f"
},
{
"path": "src/lib/blob-uploader.ts",
"chars": 5162,
"preview": "import assert = require(\"assert\");\nimport { readdir } from \"fs-extra\";\nimport * as path from \"path\";\n\nimport { Logger, l"
},
{
"path": "src/lib/common.ts",
"chars": 2450,
"preview": "import { ensureDir } from \"fs-extra\";\n\nimport { readJson, writeJson } from \"../util/io\";\nimport { joinPaths } from \"../u"
},
{
"path": "src/lib/definition-parser-worker.ts",
"chars": 924,
"preview": "import assert = require(\"assert\");\nimport process = require(\"process\");\n\nimport { getLocallyInstalledDefinitelyTyped } f"
},
{
"path": "src/lib/definition-parser.test.ts",
"chars": 3779,
"preview": "// tslint:disable:object-literal-key-quotes\n\nimport { createMockDT } from \"../mocks\";\n\nimport { getTypingInfo } from \"./"
},
{
"path": "src/lib/definition-parser.ts",
"chars": 23714,
"preview": "import { parseHeaderOrFail, TypeScriptVersion } from \"definitelytyped-header-parser\";\nimport * as ts from \"typescript\";\n"
},
{
"path": "src/lib/module-info.test.ts",
"chars": 4787,
"preview": "import * as ts from \"typescript\";\n\nimport { Dir, InMemoryDT } from \"../get-definitely-typed\";\nimport { createMockDT } fr"
},
{
"path": "src/lib/module-info.ts",
"chars": 16785,
"preview": "import assert = require(\"assert\");\nimport * as path from \"path\";\nimport * as ts from \"typescript\";\n\nimport { FS } from \""
},
{
"path": "src/lib/npm-client.ts",
"chars": 8842,
"preview": "import assert = require(\"assert\");\nimport { ensureFile, pathExists } from \"fs-extra\";\nimport RegClient = require(\"npm-re"
},
{
"path": "src/lib/package-publisher.ts",
"chars": 3498,
"preview": "import assert = require(\"assert\");\nimport { TypeScriptVersion } from \"definitelytyped-header-parser\";\n\nimport { Logger }"
},
{
"path": "src/lib/packages.test.ts",
"chars": 1825,
"preview": "import { createMockDT } from \"../mocks\";\n\nimport { getTypingInfo } from \"./definition-parser\";\nimport { TypingsVersions "
},
{
"path": "src/lib/packages.ts",
"chars": 19897,
"preview": "import assert = require(\"assert\");\nimport { AllTypeScriptVersion, Author, TypeScriptVersion } from \"definitelytyped-head"
},
{
"path": "src/lib/rolling-logs.ts",
"chars": 971,
"preview": "import BlobWriter, { readBlob } from \"./azure-container\";\n\nexport default class RollingLogs {\n static async create(na"
},
{
"path": "src/lib/secrets.ts",
"chars": 3698,
"preview": "import { AuthenticationContext } from \"adal-node\";\nimport { KeyVaultClient, KeyVaultCredentials } from \"azure-keyvault\";"
},
{
"path": "src/lib/settings.ts",
"chars": 1675,
"preview": "import { readFileSync } from \"fs\";\nimport { join as joinPaths } from \"path\";\n\n/** URL of the NPM registry to upload to. "
},
{
"path": "src/lib/versions.test.ts",
"chars": 1200,
"preview": "import { Semver } from \"./versions\";\n\ndescribe(Semver, () => {\n it(\"returns a formatted description\", () => {\n "
},
{
"path": "src/lib/versions.ts",
"chars": 5804,
"preview": "import { Logger } from \"../util/logging\";\nimport { assertDefined, best, intOfString } from \"../util/util\";\n\nimport { rea"
},
{
"path": "src/lib/webhook-server.ts",
"chars": 5810,
"preview": "import { createHmac, timingSafeEqual } from \"crypto\";\nimport { createServer, IncomingMessage, Server, ServerResponse } f"
},
{
"path": "src/make-server-run.ts",
"chars": 1100,
"preview": "import * as yargs from \"yargs\";\n\nimport { getSecret, Secret } from \"./lib/secrets\";\nimport { sourceBranch } from \"./lib/"
},
{
"path": "src/mocks.ts",
"chars": 7188,
"preview": "import { parseHeaderOrFail } from \"definitelytyped-header-parser\";\n\nimport { Dir, FS, InMemoryDT } from \"./get-definitel"
},
{
"path": "src/parse-definitions.test.ts",
"chars": 1409,
"preview": "import { createMockDT } from \"./mocks\";\nimport parseDefinitions from \"./parse-definitions\";\nimport { loggerWithErrors } "
},
{
"path": "src/parse-definitions.ts",
"chars": 3313,
"preview": "import * as yargs from \"yargs\";\n\nimport { FS, getDefinitelyTyped } from \"./get-definitely-typed\";\nimport { Options, writ"
},
{
"path": "src/publish-packages.ts",
"chars": 7793,
"preview": "import appInsights = require(\"applicationinsights\");\nimport * as yargs from \"yargs\";\n\nimport { getDefinitelyTyped } from"
},
{
"path": "src/publish-registry.ts",
"chars": 12073,
"preview": "import assert = require(\"assert\");\nimport { emptyDir } from \"fs-extra\";\nimport * as yargs from \"yargs\";\n\nimport { FS, ge"
},
{
"path": "src/test-get-secrets.ts",
"chars": 459,
"preview": "// Run `node ./bin/test-get-secrets.js` to test that we can fetch secrets from Azure Keyvault\n\nimport { allSecrets, getS"
},
{
"path": "src/tester/get-affected-packages.test.ts",
"chars": 1568,
"preview": "import { AllPackages, NotNeededPackage, TypesDataFile } from \"../lib/packages\";\nimport { createTypingsVersionRaw, testo "
},
{
"path": "src/tester/get-affected-packages.ts",
"chars": 4018,
"preview": "import { AllPackages, formatDependencyVersion, getMangledNameForScopedPackage, PackageBase, PackageId, TypingsData } fro"
},
{
"path": "src/tester/test-runner.test.ts",
"chars": 5593,
"preview": "import { NpmInfo } from \"../lib/npm-client\";\nimport { AllPackages, NotNeededPackage, TypesDataFile } from \"../lib/packag"
},
{
"path": "src/tester/test-runner.ts",
"chars": 15961,
"preview": "import assert = require(\"assert\");\nimport { existsSync, readFileSync } from \"fs\";\nimport { pathExists, remove } from \"fs"
},
{
"path": "src/tester/test.ts",
"chars": 1419,
"preview": "import yargs = require(\"yargs\");\n\nimport checkParseResults from \"../check-parse-results\";\nimport { clean } from \"../clea"
},
{
"path": "src/types/adal-node.d.ts",
"chars": 379,
"preview": "export class AuthenticationContext {\n constructor(authorization: string);\n\n acquireTokenWithClientCredentials(\n "
},
{
"path": "src/types/azure-keyvault.d.ts",
"chars": 750,
"preview": "type AuthenticatorCallback = (error: Error | null | undefined, authorization?: string) => void;\nexport class KeyVaultCre"
},
{
"path": "src/types/fstream.d.ts",
"chars": 252,
"preview": "export function Reader(options: ReaderOptions): NodeJS.ReadableStream;\ninterface ReaderOptions {\n path: string;\n t"
},
{
"path": "src/types/npm-registry-client.d.ts",
"chars": 1304,
"preview": "// Definitions transcribed from https://github.com/npm/npm-registry-client\ndeclare class RegClient {\n constructor(con"
},
{
"path": "src/upload-blobs.ts",
"chars": 266,
"preview": "import uploadBlobsAndUpdateIssue from \"./lib/blob-uploader\";\nimport { currentTimeStamp, logUncaughtErrors } from \"./util"
},
{
"path": "src/util/io.ts",
"chars": 5032,
"preview": "import {\n readFile as readFileWithEncoding,\n readFileSync as readFileWithEncodingSync,\n stat,\n writeFile as "
},
{
"path": "src/util/logging.ts",
"chars": 3460,
"preview": "import { ensureDir } from \"fs-extra\";\n\nimport { logDir } from \"../lib/settings\";\nimport { joinPaths } from \"../util/util"
},
{
"path": "src/util/progress.ts",
"chars": 2900,
"preview": "import charm = require(\"charm\");\n\nexport interface Options {\n /** Text to display in front of the progress bar. */\n "
},
{
"path": "src/util/test.ts",
"chars": 1035,
"preview": "import { License, PackageId, TypingsVersionsRaw } from \"../lib/packages\";\n\nexport function testo(o: { [s: string]: () =>"
},
{
"path": "src/util/tgz.ts",
"chars": 2417,
"preview": "import { createWriteStream } from \"fs\";\nimport { FStreamEntry, Reader } from \"fstream\";\nimport { Pack } from \"tar\";\nimpo"
},
{
"path": "src/util/util.ts",
"chars": 23980,
"preview": "import assert = require(\"assert\");\nimport { ChildProcess, exec as node_exec, fork } from \"child_process\";\nimport * as cr"
},
{
"path": "src/validate.ts",
"chars": 5667,
"preview": "import { mkdirp, remove } from \"fs-extra\";\nimport * as yargs from \"yargs\";\n\nimport { FS, getDefinitelyTyped } from \"./ge"
},
{
"path": "src/webhook.ts",
"chars": 1543,
"preview": "import appInsights = require(\"applicationinsights\");\nimport * as yargs from \"yargs\";\n\nimport { Options } from \"./lib/com"
},
{
"path": "tsconfig.json",
"chars": 558,
"preview": "{\n \"include\": [\"src\"],\n \"compilerOptions\": {\n \"module\": \"commonjs\",\n \"strict\": true,\n \"declar"
},
{
"path": "tslint.json",
"chars": 2059,
"preview": "{\n \"extends\": \"tslint:all\",\n \"rules\": {\n \"arrow-parens\": [true, \"ban-single-arg-parens\"],\n \"comment-"
}
]
About this extraction
This page contains the full source code of the microsoft/types-publisher GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 70 files (285.3 KB), approximately 68.4k tokens, and a symbol index with 502 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.