Support customization process part2 #1967

MaryGao · 2023-08-11T02:19:13Z

fixes #1811 #1958

[Part 2] In this pr we would cover:

Generate modular version for package.json and tsconfig.json
Generate metadata if relevant files are absent and not generate if existing
Refresh OpenAI to the latest tsp

Please notice the change would only apply to typespec as input. For swagger if needed we could have another pr.

…-process-part2

MaryGao · 2023-08-14T02:34:54Z

packages/autorest.typescript/test/rlcIntegration/generated/bodyFileRest/package.json

    "clean": "rimraf dist dist-browser dist-esm test-dist temp types *.tgz *.log",
    "execute:samples": "echo skipped",
    "extract-api": "rimraf review && mkdirp ./review && api-extractor run --local",
-    "format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"*.{js,json}\"  \"samples-dev/**/*.ts\"",
+    "format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"*.{js,json}\" ",


Small fix when there is no sample folder included.

is this trying to completely ignore the samples-dev folder?

This is the small fix, sample folder should be ignored if there is no samples generated.

Should not be ignored if there is samples generated.

autorest.typescript/packages/autorest.typescript/test/smoke/generated/agrifood-data-plane/package.json

Line 34 in 2b5698f

"format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"*.{js,json}\" \"test/**/*.ts\" \"samples-dev/**/*.ts\"",

can you point to me where you add this check in the codegen side ?

https://github.com/Azure/autorest.typescript/pull/1967/files#diff-c986168a4919c6df1a665149746b826e4876f0a438955d39d098852eec9abc53R74-R76

MaryGao · 2023-08-14T05:23:46Z

Below cases are covered in our smoke/integration test(T - true, F - false, U - undefined):

generateMetadata	generateTest	metaGenerated?	testGenerated?
T	T	Y	Y
T	F	Y	N
T	U	Y	Y(if absent) N(if existing)
F	F	N	N
F	T	N	Y
F	U	Y	Y(if absent) N(if existing)
U	U	Y(if absent) N(if existing)	Y(if absent) N(if existing)
U	T	Y(if absent) N(if existing)	Y
U	F	Y(if absent) N(if existing)	N

qiaozha

Not sure if we by default geenrateTest and generateSample is good or not ? given the sample quality and we might have problem to compiling the code in CI.

qiaozha · 2023-08-14T06:39:34Z

packages/autorest.typescript/test/rlcIntegration/generated/bodyFileRest/package.json

    "clean": "rimraf dist dist-browser dist-esm test-dist temp types *.tgz *.log",
    "execute:samples": "echo skipped",
    "extract-api": "rimraf review && mkdirp ./review && api-extractor run --local",
-    "format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"*.{js,json}\"  \"samples-dev/**/*.ts\"",
+    "format": "prettier --write --config ../../../.prettierrc.json --ignore-path ../../../.prettierignore \"src/**/*.ts\" \"*.{js,json}\" ",


is this trying to completely ignore the samples-dev folder?

packages/typespec-test/test/loadtesting_modular/generated/typespec-ts/tsconfig.json

packages/typespec-ts/src/index.ts

qiaozha · 2023-08-14T07:12:27Z

packages/rlc-common/src/metadata/buildApiExtractorConfig.ts

-  const { generateTest, packageDetails } = model.options || {};
+  let { generateTest, packageDetails, isModularLibrary } = model.options || {};
+  // Take the undefined as true by default
+  generateTest = generateTest === true || generateTest === undefined;


Can we have some automatical check if the tests exist or not ?

We do this automatical check in emitter side, https://github.com/Azure/autorest.typescript/pull/1967/files#diff-3d0e467e1b9edbc8377b86fbeaac71ce5bc0ddc8260bee5e56a777360cf19398R236-R242.

In common layer we also need to check if we need to generate test relevant stuff so we have the dependency of generateTest.

qiaozha · 2023-08-14T07:26:14Z

packages/rlc-common/src/metadata/buildApiExtractorConfig.ts

  const project = new Project();
  const config = {
    $schema:
      "https://developer.microsoft.com/json-schemas/api-extractor/v7/api-extractor.schema.json",
-    mainEntryPointFilePath: `./types${generateTest ? "/src" : ""}/index.d.ts`,
+    mainEntryPointFilePath: `./types${
+      generateTest || isModularLibrary ? "/src" : ""


why isModularLibrary can have impact on this mainEntryPointFilePath ?

For modular the generated index.d.ts would be in types/src folder.

even if we don't generate tests and samples ?

Yes, this is because in modular tsconfig.json we by default add below option which would always intro a src folder in dist-ems folder.

"rootDir": "."

https://www.typescriptlang.org/tsconfig#rootDir

MaryGao · 2023-08-15T03:28:21Z

Not sure if we by default geenrateTest and generateSample is good or not ? given the sample quality and we might have problem to compiling the code in CI.

The default behavior of generateSample(not generating samples) is not changed, I enable the generateTest if this config is undefined and there is no test folder. And I prefer to turn on this option because it could help add testing relevant configs and sample codes to allow customer grow up.

MaryGao · 2023-08-15T03:30:12Z

packages/typespec-ts/src/index.ts

+    const hasPackageFile = await existsSync(
+      join(dpgContext.generationPathDetail?.metadataDir ?? "", "package.json")
+    );
+    const shouldGenerateMetadata =


Here is the emitter logic to detect if we need to generate metadata.

MaryGao · 2023-08-15T03:30:26Z

packages/typespec-ts/src/index.ts

+    const hasTestFolder = await fsextra.pathExists(
+      join(dpgContext.generationPathDetail?.metadataDir ?? "", "test")
+    );
+    const shouldGenerateTest =


Here is the emitter logic to detect if we need to generate test files.

…-process-part2

MaryGao · 2023-08-15T04:53:51Z

packages/typespec-ts/src/modular/helpers/operationHelpers.ts

@@ -718,6 +719,8 @@ function deserializeResponseValue(
      return required
        ? `new Date(${restValue})`
        : `${restValue} !== undefined? new Date(${restValue}): undefined`;
+    case "combined":
+      return `${restValue} as any`;


Temp workaround for this issue #1921

MaryGao · 2023-08-15T04:54:35Z

packages/typespec-test/test/openai_modular/generated/typespec-ts/src/api/operations.ts

-            error: !p.error ? undefined : p.error,
-          })),
+          created: new Date(result.body.result?.["created"]),
+          data: result.body.result?.["data"] as any,


Workaround: #1921

MaryGao added 30 commits August 8, 2023 15:00

Update the code structure

9d5881a

Remove commented out codes

7207b5d

Code refactor

2b00178

Use the calculated generation dir

ad35497

Update to use the generation path for modular

a2fc5d6

Remove comments

a20ce42

Fix lint issue

89ad9fc

Update the generated folder with src prefix

94e79ec

Add sources folder to trigger the smoke test

890aab5

Change the copy info

b4080b2

Merge branch 'main' into support-customization-process-part2

c03c07a

Always generate metadata in rlc-common

b8d60f6

Refactor code structure for metadata generation

ad14bd0

Add comments

b58bc6a

Rename the generated code

c94ef20

Override the package.json and tsconfig as modular one

8ec4ab6

update the files

1011af8

Merge remote-tracking branch 'origin/main' into support-customization…

ecd3e24

…-process-part2

Resolve conflicts

3bce460

Update the test cases

5b7f99a

Fix

47583d8

Refresh smoke testing

6f5ae84

Regenerate integration testing

0e2793c

Fix the browser file missing issue

d2477cc

Refresh again

d377cc0

Update the files in integration testing

4fb4808

Fix lint issues

298d5cc

Update the generation logic in autorest

89e8f72

Try to fix the folder creation error

cc25f84

Revert changes in autorestOptions

e56e944

MaryGao added 2 commits August 13, 2023 15:34

Revert changes in generateRestLevel

ecea2c0

Regenerate the code in autorest

4f7c3cb

MaryGao marked this pull request as ready for review August 13, 2023 08:13

MaryGao requested review from sarangan12 and joheredi as code owners August 13, 2023 08:13

update package.json files

01cf8d7

MaryGao commented Aug 14, 2023

View reviewed changes

Update the test cases

467becb

Revert changes

2b5698f

MaryGao requested a review from qiaozha August 14, 2023 05:41

qiaozha reviewed Aug 14, 2023

View reviewed changes

MaryGao mentioned this pull request Aug 14, 2023

Broken generation of @azure/openai #1965

Closed

7 tasks

MaryGao added 2 commits August 15, 2023 11:08

Refactor to remove the program param

d5f5e7b

Rename the param name to emitterOptions

e22c75e

MaryGao requested a review from qiaozha August 15, 2023 03:28

MaryGao commented Aug 15, 2023

View reviewed changes

MaryGao added 6 commits August 15, 2023 11:31

Merge remote-tracking branch 'origin/main' into support-customization…

bdb342d

…-process-part2

Resolve the conflicts

8c188b8

Fix build failure

85cb4f5

Merge remote-tracking branch 'origin/main' into support-customization…

0b76416

…-process-part2

Add the case for combined as any to fix openai issue

8076f8c

Refresh the open ai to the latest tsp

088c3f9

MaryGao commented Aug 15, 2023

View reviewed changes

Enable testing files for openai

930ab6a

qiaozha approved these changes Aug 15, 2023

View reviewed changes

MaryGao merged commit e3cc0f4 into Azure:main Aug 15, 2023
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support customization process part2 #1967

Support customization process part2 #1967

MaryGao commented Aug 11, 2023 •

edited

Loading

MaryGao Aug 14, 2023

qiaozha Aug 14, 2023

MaryGao Aug 15, 2023

qiaozha Aug 15, 2023

MaryGao Aug 15, 2023

MaryGao commented Aug 14, 2023 •

edited

Loading

qiaozha left a comment

qiaozha Aug 14, 2023

qiaozha Aug 14, 2023

MaryGao Aug 15, 2023 •

edited

Loading

qiaozha Aug 14, 2023

MaryGao Aug 15, 2023

qiaozha Aug 15, 2023

MaryGao Aug 15, 2023 •

edited

Loading

MaryGao commented Aug 15, 2023

MaryGao Aug 15, 2023

MaryGao Aug 15, 2023

MaryGao Aug 15, 2023

MaryGao Aug 15, 2023

Support customization process part2 #1967

Support customization process part2 #1967

Conversation

MaryGao commented Aug 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaryGao commented Aug 14, 2023 • edited Loading

qiaozha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaryGao Aug 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaryGao Aug 15, 2023 • edited Loading

Choose a reason for hiding this comment

MaryGao commented Aug 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaryGao commented Aug 11, 2023 •

edited

Loading

MaryGao commented Aug 14, 2023 •

edited

Loading

MaryGao Aug 15, 2023 •

edited

Loading

MaryGao Aug 15, 2023 •

edited

Loading