Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add preview Log Analytics query extension #250

Merged
merged 23 commits into from
Jul 27, 2018

Conversation

alexeldeib
Copy link
Contributor

@alexeldeib alexeldeib commented Jul 26, 2018


This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines

  • Have you run ./scripts/ci/test_static.sh locally? (pip install pylint flake8 required)
  • Have you run python scripts/ci/test_index.py -q locally?

For new extensions:

@alexeldeib alexeldeib changed the title Add preview Log Analytics extension Add preview Log Analytics query extension Jul 26, 2018
@azuresdkci
Copy link

If this PR is for a new extension or change to an existing extension, use the following to try out the changes in this PR:

docker run -it microsoft/azure-cli:latest
export EXT=<NAME>
pip install --upgrade --target ~/.azure/cliextensions/$EXT "git+https://github.com/alexeldeib/azure-cli-extensions.git@loganalytics#subdirectory=src/$EXT&egg=$EXT"

@alexeldeib
Copy link
Contributor Author

Hi -- Per offline discussion with @mayurid @williexu, reintroducing Log Analytics query functionality as an extension.

I had some issues running/recording the tests after migrating from module-style to extension-style. Specifically I can't import azure.cli.testsdk package. I assumed this was due to the namespacing issue referenced in the faq, but I see many other packages using this in their tests? Do I need to set up my environment somehow differently to make this work properly?

@williexu
Copy link
Contributor

williexu commented Jul 26, 2018

@alexeldeib You should be able to import from azure.cli.testsdk. Many of the other extensions do the same. Are you following these instructions to test your extension?

Also, can you introduce the extension commands under the monitor command-group (i.e. 'az monitor log-analytics query')? and address CI

@alexeldeib
Copy link
Contributor Author

Yes, I'm using separate virtualenvs to test. Is there possibly a related package I have installed globally that could be messing with the namespacing? For good measure I ran pytest and the same issues occurs with all the extensions, so it's definitely something in my env..

I'll fix the other issues.

Copy link
Contributor

@williexu williexu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, rebase to get updates from master, address comments and add yourself as codeowner :)

src/index.json Outdated
@@ -951,6 +951,52 @@
"version": "0.9.0"
}
}
],
"loganalytics": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the command will be az monitor log-analytics(with the dash), you may want to rename to log-analytics for more clarity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hosting on PyPi and we already have the azure-loganalytics python sdk there, so I was going to follow the azure batch style naming and go with azure-loganalytics-cli-extension. I see that test_index.py validates the package name against the .dist-info metadata.json, so would that (azure-loganalytics-cli-extension) be fine here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave the name as log-analytics as this is the name users will see when they run the az extension list-available command. This being an extension for azure-cli is implied implicitly.

Batch and others were added before these naming conventions were made. The hosting location does not matter for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

src/index.json Outdated
"metadata_version": "2.0",
"name": "loganalytics",
"summary": "Support for Azure Log Analytics query functionality.",
"version": "1.0.0-Preview-1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preview version will start with 0.1.0, please change the version.

1.0.0-Preview-1
++++++++++++++++++

* Initial release.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start with version 0.1.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

edit: replied to stale comment

src/index.json Outdated
"description": "DESCRIPTION.rst"
},
"project_urls": {
"Home": "https://github.com/Azure/azure-cli-extensions"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to point to "https://github.com/Azure/azure-cli-extensions/tree/master/src/log-analytics" so that people will be directed straight towards your readme.


# pylint: disable=line-too-long

helps['loganalytics query'] = """
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change to reflect the command

client_factory=loganalytics_data_plane_client
)

with self.command_group('monitor loganalytics', loganalytics_sdk) as g:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Command must be monitor log-analytics for consistency with CLI commands.

)

with self.command_group('monitor loganalytics', loganalytics_sdk) as g:
g.command('query', 'execute_query')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use g.custom_command() to point directly at the custom namespace defined in your loader, no need to reference the same namespace again and will open up the possibility for specifying other sdks if needed.

from codecs import open
from setuptools import setup, find_packages

VERSION = "1.0.0-Preview-2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.1.0

@alexeldeib
Copy link
Contributor Author

Fixed up my python environment and most of the remaining issues, I'll keep an eye on CI. Let me know if you see anything else that sticks out.


# pylint: disable=line-too-long

helps['monitor loganalytics query'] = """
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log-analytics



def load_arguments(self, _):
with self.argument_context('loganalytics') as c:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

monitor log-analytics query

def load_arguments(self, _):
with self.argument_context('loganalytics') as c:
c.argument('workspace', options_list=['--workspace', '-w'], help='GUID of the Log Analytics Workspace', required=True)
c.argument('kql', help='Query to execute over Log Analytics data.', required=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need the required=True for workspace or kql; they will be implied from the command signature in execute_query

self.check('tables[0].rows[0][0]', 'TenantId')
])
query_result = self.cmd('az loganalytics query -w cab864ad-d0c1-496b-bc5e-4418315621bf --kql "Heartbeat | getschema"').get_output_in_json()
assert len(query_result['tables'][0]['rows']) == 29
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test looks stale. Can you run your test again? that should also pick up some of the other problems i saw.

license='MIT',
author='Ace Eldeib',
author_email='aleldeib@microsoft.com',
url='https://github.com/Azure/azure-cli-extensions/tree/master/src/log-analytics',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't I just change the directory to match? Since we're adding the dash everywhere, may as well be consistent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes pls

Copy link
Contributor

@williexu williexu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last thing, remove the indicated files leftover from testing

@@ -0,0 +1,3 @@
[
"azext_loganalytics/tests/latest/test_loganalytics_commands.py::LogAnalyticsDataClientTests::test_query"
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

@@ -0,0 +1 @@
{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

@alexeldeib
Copy link
Contributor Author

alexeldeib commented Jul 27, 2018

Thanks for all the help cleaning this up! One minor thing, I want to see if I can come up with a better name for --kql parameter. Normally we use "query" but it's used globally 😄 How about --analytics-query?

@williexu
Copy link
Contributor

williexu commented Jul 27, 2018

@alexeldeib --analytics-query sounds good to me. Do you want to include that with this PR?

@williexu
Copy link
Contributor

williexu commented Jul 27, 2018

@alexeldeib for futures PRs, I'd recommend resolving all issues with the extension before updating the index. That will save you from having to build a new wheel file, upload to pypi and update the index/history/version multiple times.

@williexu williexu merged commit 89518d0 into Azure:master Jul 27, 2018
necusjz pushed a commit to necusjz/azure-cli-extensions that referenced this pull request Sep 1, 2023
necusjz pushed a commit to necusjz/azure-cli-extensions that referenced this pull request Sep 1, 2023
wangzelin007 pushed a commit that referenced this pull request Sep 4, 2023
* Kubernetes Data Protection Extension CLI (#173)

* First draft for Data Protection K8s backup extension (Pending internal review)

* Removing tracing

* Minor changes to improve azdev style

* Internal PR review feedback

Co-authored-by: Rishabh Raj <rishraj@microsoft.com>

* {AKS - ARC} fix: Update DCR creation to Clusters resource group instead of workspace (#175)

* fix: Update DCR creation to Clusters resource group instead of workspace

* .

* .

* casing check

* Add self-signed cert to fix PR gate for azureml extension

* adding the api version to the operation definition in the client factory

* bump k8s-extension version to 1.3.6

* adding tests for all 4 extension types calls

* adding to test config file

* updating the api version for extension types to be the correct version expected by the service

* add test case for flux extension (#184)

* bump k8s-extension version to 1.3.6

* bump k8s-extension version to 1.3.6

* adding upstream test for extension types

* updating history.rst

* [Dapr] Prompt user for existing Dapr installation during extension create (#188)

* Add more validations and user prompt for existing installation scenario

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add Dapr test'

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Handle stateful set

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update default handling

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix HA handling

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add placement service todo

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add non-interactive mode

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Reset configuration for StatefulSet during k8s upgrade

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Retrigger tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add changes to manage ha and placement params

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update message

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* nits

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* bump k8s-extension version to 1.3.7

* [Dapr] Disable applying CRDs during a downgrade (#193)

* Add logging

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update log

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Revert applyCrds when not downgrading

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update logic for removing hooks.applyCrds

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Revert logic

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Handle explicit hooks configuration

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update comment

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* re-trigger pipeline

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* ContainerInsights extension -  Add dataCollectionSettings configuration settings (#200)

* data collection settings

* add support for dataCollectionSettings

* fix indention

* avoid duplicate use of json loads

* remove whitespaces

* fix pr feedback

* Upgrade Python version from 3.6 to 3.7 (#203)

* Upgrade Python version from 3.6 to 3.10

Upgrade to 3.10 for the job that runs Wheel, PyLint, Flake, etc., since 3.6 is not supported anymore by hosted-agent-software.

* Upgrade to Python 3.10 from 3.6

Upgrade to 3.10 as 3.6 is not supported

* Switch PyLink to 1.9.4

Switch PyLink to 1.9.4 from 1.9.5, as 1.9.5 is not supported with Python 3.10

* Use Python 3.7 for Static Analysis

Use 3.7, as 3.10 does not support certain properties used by astpeephole.py that is used by Static Analysis tools

* Try unpinned version of PyLint

PyLint 1.9.5 doesn't work with Python 3.7.  So, trying to see if it automatically pulls the latest compatible version.

* Run pylint as a separate command

* Update pylintrc (#204)

* Update pylintrc

* Update k8s-custom-pipelines.yml

* Disable PyLint (#205)

Disable PyLint for now, as the new version has breaking changes and requires lot more fixes

* Disable PyLint on CI scripts

* Fixes for script errors

* Upgrade Static Analysis Python version

Upgrade the Python version for Static Analysis to 3.10, from 3.7, now that PyLint is disabled

* Try 3.9, as 3.10 has breaking changes for Flake8

* Remove version pinning for flake8

Try Python 3.10, without pinning flake8 to a version

* Update k8s-custom-pipelines.yml

* Use Python 3.8.1 & flake8 6.0.0

* Use Python 3.8 instead of 3.8.1

* Update k8s-custom-pipelines.yml

* Update .flake8

Update to reflect breaking change in flake8 6.0

* Update source_code_static_analysis.py

Scope static analysis tools to only k8s-extension module's source in our branch.

* Update k8s-custom-pipelines.yml

* Update k8s-custom-pipelines.yml

* Update k8s-custom-pipelines.yml

* Update pool name in StaticAnalysis

To mirror what is in main of azure-cli-extensions

* Update k8s-custom-pipelines.yml

* Fix indentation

* Update k8s-custom-pipelines.yml

* Update k8s-custom-pipelines.yml

* Revert changes

* Revert changes

* Revert changes to source_code_static_analysis.py

* Update source_code_static_analysis.py

* Revert changes

* Use Ubuntu 20.4 for BuiltTestPublish stage

* Switch to ubuntu-20.04 from latest

Co-authored-by: Rishik Hombal <hombalrishik@gmail.com>

* [Dapr] Do not apply CRD hook when version is unchanged or auto-upgrade is being disabled (#201)

* Update logic

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* re-trigger pipeline

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* re-trigger pipeline

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
Co-authored-by: NarayanThiru <nanthi@microsoft.com>

* add dummy key for amalogs as well

* bump k8s-extension version to 1.3.8

* Adding GA api version 2022-11-01 exposing isSystemExtension and support for plan info

* Seperate args for plan name, product and publisher

* updating cassete file

* updating HISTORY.rst

* Deprecate longer parameter names when accepting config settings (#213)

Co-authored-by: deeksha345 <34255011+deeksha345@users.noreply.github.com>

* Release 1.3.9

* make containerinsights dcr name consistent (#211)

Co-authored-by: Bavneet Singh <33008256+bavneetsingh16@users.noreply.github.com>

* [Dapr] Update version comparison logic to use semver based comparison (#219)

* Update semver comparison

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add log

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

---------

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* bump k8s-extension version to 1.4.0 (#220)

* Revert "bump k8s-extension version to 1.4.0 (#220)" (#222)

This reverts commit ffb8a95.

* [k8s-extension] Update extension CLI to v1.4.0

* update release history

* fix openservice mesh cli testcase issue

* Zetia/fix ssl secret flag (#224)

* fix bug: update operation doesn't respect sslSecret parameter

* fix bug: update operation doesn't respect sslSecret parameter

* fix typo

* feat: public preview support for microsoft.azuremonitor.containers.metrics in ARC clusters (managed prometheus) (#227)

* remove redundant extension test (#230)

* ci MSI default for arc cluster (#231)

* bump k8s-extension version to 1.4.2

* ContainerInsights extension - Extend dataCollectionSettings config settings with streams field (#232)

* extend containerinsights datacollection settings with streams field

* bug fix

* fix lint issues

* fix pr feedback

* fix pr feedback

* fix lint error

* Generated files for 2023-05-01-preview

* Support for 2023-05-01-preview

* Rename get to show

* Added ExtensionType api test cases

* ContainerInsights extension - Extend dataCollectionSettings with containerlogv2 (#237)

* Fix for Liniting issues

* Fixing test cases

* comment failing test cases

* [k8s-extension] add kind tag in DCR creation (#240)

* Use semver package (#241)

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Reverting commented test cases

* Add support to skip provisioning of prerequisites for Azure Monitor K8s extensions (#234)

* {ARC} fix: update logic to sanitize cluster name for dc* objects (#242)

* Fix osm-arc version check for CI tags (#244)

Signed-off-by: nshankar <nshankar@microsoft.ghe.com>
Co-authored-by: nshankar <nshankar@microsoft.ghe.com>

* New cassette file

* Remove unused propeties from table format

* bump k8s-extension version 1.4.3

* Add old commands back with deprecated status

* Fix linting issues

* Reverting changes for extensions type api

* change the location for test runs and arc clusters

* [k8s-extension] create new cli release - v1.4.3 (#250)

* Revert "[k8s-extension] create new cli release - v1.4.3 (#250)" (#251)

This reverts commit 584815d.

* [k8s-extension] Update extension CLI to v1.4.3

* Drop relay sdk (#254)

* update readme

* remove useless snippets (#256)

* [k8s-extension] Update extension CLI to v1.4.4

---------

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
Signed-off-by: nshankar <nshankar@microsoft.ghe.com>
Co-authored-by: Rishabh Raj <rishabhstpaul@gmail.com>
Co-authored-by: Rishabh Raj <rishraj@microsoft.com>
Co-authored-by: bragi92 <kadubey@microsoft.com>
Co-authored-by: Yue Yu <yuyu3@microsoft.com>
Co-authored-by: Deeksha Sharma <deesharma@microsoft.com>
Co-authored-by: deeksha345 <34255011+deeksha345@users.noreply.github.com>
Co-authored-by: Shubham Sharma <shubhash@microsoft.com>
Co-authored-by: Bavneet Singh <bavneetsingh@microsoft.com>
Co-authored-by: Ganga Mahesh Siddem <gangams@microsoft.com>
Co-authored-by: NarayanThiru <nanthi@microsoft.com>
Co-authored-by: Rishik Hombal <hombalrishik@gmail.com>
Co-authored-by: Amol Agrawal <amagraw@microsoft.com>
Co-authored-by: Amol Agrawal <pfrcks@gmail.com>
Co-authored-by: Arif Lakhani <ariflakhani@microsoft.com>
Co-authored-by: Arif-lakhani <ariflakhani7786@gmail.com>
Co-authored-by: Zeliang Tian <83852443+zetiaatgithub@users.noreply.github.com>
Co-authored-by: Long Wan <wanlonghenry@gmail.com>
Co-authored-by: ms-hujia <48512395+ms-hujia@users.noreply.github.com>
Co-authored-by: Niranjan Shankar <nshankar@microsoft.com>
Co-authored-by: nshankar <nshankar@microsoft.ghe.com>
Co-authored-by: necusjz <necusjz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants