{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":366482973,"defaultBranch":"main","name":"anomaly-detection-1","ownerLogin":"kaituo","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2021-05-11T18:48:17.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/5303417?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1726619907.0","currentOid":""},"activityList":{"items":[{"before":"447cabdb5bbec7872d4e02863c246466922b346f","after":"7feb12f488fde185e8c3b565fb34fb1381a55433","ref":"refs/heads/proto","pushedAt":"2024-09-18T18:17:40.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Support forecast tasks in profile API; enable index field modifications\n\nThis PR adds support for forecast task targets in profile API and enables modification of categorical and custom result index fields\n* Support forecast task targets in profile API: Previously, the profile API returned empty results for forecast task targets. This update adds support for them. For details, refer to ForecastTask.\n* Allow modification of categorical and custom result index fields in forecasting: These fields are not editable in Anomaly Detection (AD). Implemented a lastUiBreakingChangeTime field in the config index to manage changes. Whenever the categorical or custom result index fields are updated, lastUiBreakingChangeTime is refreshed. The UI will not display results before this time to prevent inconsistencies. For details, refer to Config and AbstractTimeSeriesActionHandler.\n\nTesting Done:\n* Added ITs for both scenarios.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Support forecast tasks in profile API; enable index field modifications"}},{"before":null,"after":"447cabdb5bbec7872d4e02863c246466922b346f","ref":"refs/heads/proto","pushedAt":"2024-09-18T00:38:27.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Support forecast tasks in profile API; enable index field modifications\n\nThis PR adds support for forecast task targets in profile API and enables modification of categorical and custom result index fields\n* Support forecast task targets in profile API: Previously, the profile API returned empty results for forecast task targets. This update adds support for them. For details, refer to ForecastTask.\n* Allow modification of categorical and custom result index fields in forecasting: These fields are not editable in Anomaly Detection (AD). Implemented a lastUiBreakingChangeTime field in the config index to manage changes. Whenever the categorical or custom result index fields are updated, lastUiBreakingChangeTime is refreshed. The UI will not display results before this time to prevent inconsistencies. For details, refer to Config and AbstractTimeSeriesActionHandler.\n\nTesting Done:\n* Added ITs for both scenarios.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Support forecast tasks in profile API; enable index field modifications"}},{"before":null,"after":"ea3c6f000e2fd620d36deee1d9c8bf86d06091ca","ref":"refs/heads/imputeBug217","pushedAt":"2024-09-13T23:24:26.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Bump BWC Version to 2.18 and Fix Bugs (#1311)\n\nThis PR includes the following updates and bug fixes:\n\n* Bump BWC Version to 2.18: Updated BWC version to 2.18 since the 2.17 branch has been cut.\n* Fix Confidence Value Exceeding 1 in RCF: Addressed a bug in RCF where the confidence value could exceed 1. Implemented a check to cap the confidence value at 1, preventing invalid confidence scores.\n* Correct Parameter Assignment in GetAnomalyDetectorTransportAction: Fixed an issue where parameter assignments within a method did not affect external variables due to Java's pass-by-value nature.\n* Fixed a bug in ResultProcessor where we were supposed to check whether the number of sent messages equals the number of received messages before starting imputation. However, the sent message count was mistakenly based on the number of pages rather than the actual number of messages.\n* Fixed a bug where we mistakenly used the total reserved memory bytes as the memory size per entity in PriorityCache.\n\nTesting done:\n* added test cases for the buggy scenarios\n* manual e2e testing\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Bump BWC Version to 2.18 and Fix Bugs (opensearch-project#1311)"}},{"before":null,"after":"b3d033bfeaa8517912c31a0660004cafab3acd89","ref":"refs/heads/imputeBug2x","pushedAt":"2024-09-13T23:21:49.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Bump BWC Version to 2.18 and Fix Bugs (#1311)\n\nThis PR includes the following updates and bug fixes:\n\n* Bump BWC Version to 2.18: Updated BWC version to 2.18 since the 2.17 branch has been cut.\n* Fix Confidence Value Exceeding 1 in RCF: Addressed a bug in RCF where the confidence value could exceed 1. Implemented a check to cap the confidence value at 1, preventing invalid confidence scores.\n* Correct Parameter Assignment in GetAnomalyDetectorTransportAction: Fixed an issue where parameter assignments within a method did not affect external variables due to Java's pass-by-value nature.\n* Fixed a bug in ResultProcessor where we were supposed to check whether the number of sent messages equals the number of received messages before starting imputation. However, the sent message count was mistakenly based on the number of pages rather than the actual number of messages.\n* Fixed a bug where we mistakenly used the total reserved memory bytes as the memory size per entity in PriorityCache.\n\nTesting done:\n* added test cases for the buggy scenarios\n* manual e2e testing\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Bump BWC Version to 2.18 and Fix Bugs (opensearch-project#1311)"}},{"before":"c637d20f22e2461f7920687335e82fba43a4bfba","after":"f6d41a007d08e5cd993cb8bfaec681556fb18ebc","ref":"refs/heads/imputeBug","pushedAt":"2024-09-13T22:34:41.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Bump BWC Version to 2.18 and Fix Bugs\n\nThis PR includes the following updates and bug fixes:\n\n* Bump BWC Version to 2.18: Updated BWC version to 2.18 since the 2.17 branch has been cut.\n* Fix Confidence Value Exceeding 1 in RCF: Addressed a bug in RCF where the confidence value could exceed 1. Implemented a check to cap the confidence value at 1, preventing invalid confidence scores.\n* Correct Parameter Assignment in GetAnomalyDetectorTransportAction: Fixed an issue where parameter assignments within a method did not affect external variables due to Java's pass-by-value nature.\n* Fixed a bug in ResultProcessor where we were supposed to check whether the number of sent messages equals the number of received messages before starting imputation. However, the sent message count was mistakenly based on the number of pages rather than the actual number of messages.\n* Fixed a bug where we mistakenly used the total reserved memory bytes as the memory size per entity in PriorityCache.\n\nTesting done:\n* added test cases for the buggy scenarios\n* manual e2e testing\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Bump BWC Version to 2.18 and Fix Bugs"}},{"before":null,"after":"c637d20f22e2461f7920687335e82fba43a4bfba","ref":"refs/heads/imputeBug","pushedAt":"2024-09-13T22:32:37.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Bump BWC Version to 2.18 and Fix Bugs\n\nThis PR includes the following updates and bug fixes:\n\n* Bump BWC Version to 2.18: Updated BWC version to 2.18 since the 2.17 branch has been cut.\n* Fix Confidence Value Exceeding 1 in RCF: Addressed a bug in RCF where the confidence value could exceed 1. Implemented a check to cap the confidence value at 1, preventing invalid confidence scores.\n* Correct Parameter Assignment in GetAnomalyDetectorTransportAction: Fixed an issue where parameter assignments within a method did not affect external variables due to Java's pass-by-value nature.\n* Fixed a bug in ResultProcessor where we were supposed to check whether the number of sent messages equals the number of received messages before starting imputation. However, the sent message count was mistakenly based on the number of pages rather than the actual number of messages.\n* Fixed a bug where we mistakenly used the total reserved memory bytes as the memory size per entity in PriorityCache.\n\nTesting done:\n* added test cases for the buggy scenarios\n* manual e2e testing\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Bump BWC Version to 2.18 and Fix Bugs"}},{"before":"d44737e14eb7fee432906ec7ed72607580c7d9c7","after":"88e88efa13fea4b0f6c60d5f1d5abc815d5c48bf","ref":"refs/heads/2.15_patch","pushedAt":"2024-09-12T21:12:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"fix bug\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"fix bug"}},{"before":null,"after":"7b50e44b609606b8def5544303485289237b7039","ref":"refs/heads/searchDataSecurity","pushedAt":"2024-09-11T22:00:51.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix Security Tests After Changes to Permissions Requirements\n\nThis PR addresses errors in security tests caused by recent changes in opensearch-project/security#4719. Previously, users needed both AD full access and source index permissions to fully utilize anomaly detection (AD). AD full access has already included all alias and mapping permissions. it was inconsistent not to include index search permission, which would otherwise force users to create an additional role. The change in the referenced PR aimed to simplify user management.\n\nDue to this change, existing security tests that relied on a user having AD full access but lacking data search permission would no longer trigger the expected search permission exception. This PR addresses that issue by creating a new user role with only AD read permission (note we didn't change ad read access permission in the referenced PR) and without source index search permission, ensuring the tests correctly validate the lack of search permissions.\n\nTesting Done:\n* Verified that previously failing security tests now pass\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix Security Tests After Changes to Permissions Requirements"}},{"before":null,"after":"d44737e14eb7fee432906ec7ed72607580c7d9c7","ref":"refs/heads/2.15_patch","pushedAt":"2024-09-10T20:03:59.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"release notes for 2.17 (#1299) (#1300)\n\n(cherry picked from commit b84dbfeaec7ef14d5e38c007d6a33477325e9551)\n\nSigned-off-by: Jackie Han \nSigned-off-by: github-actions[bot] \nCo-authored-by: github-actions[bot] ","shortMessageHtmlLink":"release notes for 2.17 (opensearch-project#1299) (opensearch-project#…"}},{"before":null,"after":"8b49399f0ff85b08527abfdb6b5073b156fd818d","ref":"refs/heads/latestTimeBug","pushedAt":"2024-09-03T19:03:00.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Correct handling of null max aggregation values in SearchResponse\n\nPreviously, when the max aggregation in SearchResponse returned a null value, it was converted to a negative number (-9223372036854775808), leading to erroneous timestamps.\n\nThis fix adds a check to ensure that only valid positive aggregation values are considered, preventing null or invalid values from being misinterpreted as negative numbers. The change applies to only timestamps. So a positive number filter is always right.\n\nTesting done:\n1. Added unit tests to cover scenarios where max aggregation returns null or negative values, ensuring the method returns Optional.empty() when the value is invalid.\n2. Added unit tests to verify correct handling of valid positive max aggregation values.\n3. Ran e2e manual testing.\n\nPreviously:\n\n```\nPOST /_plugins/_anomaly_detection/detectors/_validate/model\n{\n \"name\": \"test-detector-13\",\n \"description\": \"Test detector\",\n \"time_field\": \"customer_birth_date\",\n \"indices\": [\"opensearch_dashboards_sample_data_ecommerce\"],\n \"feature_attributes\": [\n {\n \"feature_name\": \"cpu\",\n \"feature_enabled\": true,\n \"aggregation_query\": {\n \"total_revenue_usd-field\": {\n \"max\": {\n \"field\": \"day_of_week_i\"\n }\n }\n }\n }\n ],\n \"detection_interval\": {\n \"period\": {\n \"interval\":1,\n \"unit\": \"Minutes\"\n }\n },\n \"window_delay\": {\n \"period\": {\n \"interval\": 1,\n \"unit\": \"Minutes\"\n }\n },\n \"category_field\": [\"currency\"]\n}\n```\n\nreturns:\n\n```\nnested: OpenSearchParseException[failed to parse date field [-9223372036854775808] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [-9223372036854775808] with format [strict_date_optional_time||epoch_millis]]]; nested: IllegalArgumentException[failed to parse date field [-9223372036854775808] with format [strict_date_optional_time||epoch_millis]]; nested: DateTimeParseException[Failed to parse with all enclosed parsers]; } opensearch-ccs-node1 | [2024-09-03T15:05:45,776][ERROR][o.o.t.r.h.ModelValidationActionHandler] [2a2cd14da04d] Failed to create search request for last data point opensearch-ccs-node1 | org.opensearch.action.search.SearchPhaseExecutionException: all shards failed opensearch-ccs-node1 | at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:770) [opensearch-3.0.0.jar:3.0.0]\n```\n\nNow the call returns:\n\n```\n{ \"model\": { \"time_field\": { \"message\": \"There isn't enough historical data found with current timefield selected.\" } } }\n```\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Correct handling of null max aggregation values in SearchResponse"}},{"before":"b6503c3ab5de26411c440e964823544ce3fe17d4","after":"065a64b50076ef13f28dc59d08bcd8bbb0f14432","ref":"refs/heads/stateBug","pushedAt":"2024-08-30T19:27:28.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix: Prevent resetting latest flag of real-time analysis when starting historical analysis\n\nThis PR addresses a bug where starting a historical analysis after a real-time analysis on the same detector caused the real-time task’s latest flag to be incorrectly reset to false by the historical run.\n\nThe fix ensures that only the latest flags of the same analysis type are reset:\n* Real-time analysis will only reset the latest flag of previous real-time analyses.\n* Historical analysis will only reset the latest flag of previous historical analyses.\n\nThis PR also updated recencyEmphasis to have a minimum value of 2, aligning with RCF requirements.\n\nTesting:\n- Added an integration test to reproduce the bug and verified the fix.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix: Prevent resetting latest flag of real-time analysis when startin…"}},{"before":"2d5aff9f857d88e07098b643390701aa9d87d255","after":"b6503c3ab5de26411c440e964823544ce3fe17d4","ref":"refs/heads/stateBug","pushedAt":"2024-08-30T19:25:43.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix: Prevent resetting latest flag of real-time analysis when starting historical analysis\n\nThis PR addresses a bug where starting a historical analysis after a real-time analysis on the same detector caused the real-time task’s latest flag to be incorrectly reset to false by the historical run.\n\nThe fix ensures that only the latest flags of the same analysis type are reset:\n* Real-time analysis will only reset the latest flag of previous real-time analyses.\n* Historical analysis will only reset the latest flag of previous historical analyses.\n\nThis PR also updated recencyEmphasis to have a minimum value of 2, aligning with RCF requirements.\n\nTesting:\n- Added an integration test to reproduce the bug and verified the fix.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix: Prevent resetting latest flag of real-time analysis when startin…"}},{"before":"7010e0665c1146abec8e3757b3a3165d11481bea","after":"2d5aff9f857d88e07098b643390701aa9d87d255","ref":"refs/heads/stateBug","pushedAt":"2024-08-30T18:54:24.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix: Prevent resetting latest flag of real-time analysis when starting historical analysis\n\nThis PR addresses a bug where starting a historical analysis after a real-time analysis on the same detector caused the real-time task’s latest flag to be incorrectly reset to false by the historical run.\n\nThe fix ensures that only the latest flags of the same analysis type are reset:\n* Real-time analysis will only reset the latest flag of previous real-time analyses.\n* Historical analysis will only reset the latest flag of previous historical analyses.\n\nThis PR also updated recencyEmphasis to have a minimum value of 2, aligning with RCF requirements.\n\nTesting:\n- Added an integration test to reproduce the bug and verified the fix.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix: Prevent resetting latest flag of real-time analysis when startin…"}},{"before":null,"after":"7010e0665c1146abec8e3757b3a3165d11481bea","ref":"refs/heads/stateBug","pushedAt":"2024-08-30T15:15:53.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix: Prevent resetting latest flag of real-time analysis when starting historical analysis\n\nThis PR addresses a bug where starting a historical analysis after a real-time analysis on the same detector caused the real-time task’s latest flag to be incorrectly reset to false by the historical run.\n\nThe fix ensures that only the latest flags of the same analysis type are reset:\n* Real-time analysis will only reset the latest flag of previous real-time analyses.\n* Historical analysis will only reset the latest flag of previous historical analyses.\n\nThis PR also updated recencyEmphasis to have a minimum value of 2, aligning with RCF requirements.\n\nTesting:\n- Added an integration test to reproduce the bug and verified the fix.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix: Prevent resetting latest flag of real-time analysis when startin…"}},{"before":"a8637d07e0bd8478fc9e98c15340c29a8260ba10","after":"815f30f444cffed15431f122efb57870c36bad64","ref":"refs/heads/missingFix","pushedAt":"2024-08-23T00:36:11.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Fix inference logic and standardize config index mapping\n\nThis commit addresses several issues and improvements in the inference logic and config index mapping:\n\n1. Fixes in RealTimeInferencer:\n\n* Previously, we checked if the last update time of the model state was within the current interval and skipped inference if it was. However, this led to excessive skipping of inference because the last update time was updated when retrieving the model state from the cache.\n* Introduced lastSeenExecutionEndTime in the model state, which specifically tracks the last time a sample was processed during inference (not training). This ensures more accurate control over when inference should be skipped.\n\n2. Consistent Naming in Config Index Mapping:\n\n* To maintain consistency across the codebase, changed defaultFill to default_fill in the Config index mapping, following the underscore naming convention used elsewhere.\n\n3. Additional Null Checks:\n\n* Added more null checks for the defaultFill field in the Config constructor to improve robustness.\n\nTesting:\n* Added a smoke test to allow the job scheduler to trigger anomaly detection inferencing, successfully reproducing and verifying the fix for item #1.* added unit tests for item #3.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Fix inference logic and standardize config index mapping"}},{"before":null,"after":"a8637d07e0bd8478fc9e98c15340c29a8260ba10","ref":"refs/heads/missingFix","pushedAt":"2024-08-22T01:27:45.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"test\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"test"}},{"before":"d614b2f593e86eb309b61a9e9ee1582170407383","after":"3d93b50770b3955519ba59a9493ca206f89273fc","ref":"refs/heads/missing_2x_2","pushedAt":"2024-08-19T20:48:33.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Add Support for Handling Missing Data in Anomaly Detection (#1274)\n\n* Add Support for Handling Missing Data in Anomaly Detection\n\nThis PR introduces enhanced handling of missing data, giving customers the flexibility to choose how to address gaps in their data. Options include ignoring missing data (default behavior), filling with fixed values (customer-specified), zeros, or previous values. These options can improve recall in anomaly detection scenarios. For example, in this forum discussion https://forum.opensearch.org/t/do-missing-buckets-ruin-anomaly-detection/16535, customers can now opt to fill missing values with zeros to maintain detection accuracy.\n\nKey Changes:\n1. Enhanced Missing Data Handling:\n\nChanged to ThresholdedRandomCutForest.process(double[] inputPoint, long timestamp, int[] missingValues) to support missing data in both real-time and historical analyses. The preview mode remains unchanged for efficiency, utilizing existing linear imputation techniques. (See classes: ADColdStart, ModelColdStart, ModelManager, ADBatchTaskRunner).\n\n2. Refactoring Imputation & Processing:\n\nRefactored the imputation process, failure handling, statistics collection, and result saving in Inferencer.\n\n3. Improved Imputed Value Reconstruction:\n\nReconstructed imputed values using existing mean and standard deviation, ensuring they are accurately stored in AnomalyResult. Added a featureImputed boolean tag to flag imputed values. (See class: AnomalyResult).\n\n4. Broadcast Support for HC Detectors:\n\nAdded a broadcast mechanism for HC detectors to identify entity models that haven’t received data in a given interval. This ensures models in memory process all relevant data before imputation begins. Single stream detectors handle this within existing transport messages. (See classes: ADHCImputeTransportAction, ADResultProcessor, ResultProcessor).\n\n5. Introduction of ActionListenerExecutor:\n\nAdded ActionListenerExecutor to wrap response and failure handlers in an ActionListener, executing them asynchronously using the provided ExecutorService. This allows us to handle responses in the AD thread pool.\n\nTesting:\nComprehensive testing was conducted, including both integration and unit tests. Of the 7135 lines added and 1683 lines removed, 4926 additions and 749 deletions are in tests, ensuring robust coverage.\n\nSigned-off-by: Kaituo Li \n\n* rebase from main\n\nSigned-off-by: Kaituo Li \n\n* add comment and remove redundant code\n\nSigned-off-by: Kaituo Li \n\n---------\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Add Support for Handling Missing Data in Anomaly Detection (opensearc…"}},{"before":null,"after":"d614b2f593e86eb309b61a9e9ee1582170407383","ref":"refs/heads/missing_2x_2","pushedAt":"2024-08-19T17:52:57.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Add Support for Handling Missing Data in Anomaly Detection (#1274)\n\n* Add Support for Handling Missing Data in Anomaly Detection\n\nThis PR introduces enhanced handling of missing data, giving customers the flexibility to choose how to address gaps in their data. Options include ignoring missing data (default behavior), filling with fixed values (customer-specified), zeros, or previous values. These options can improve recall in anomaly detection scenarios. For example, in this forum discussion https://forum.opensearch.org/t/do-missing-buckets-ruin-anomaly-detection/16535, customers can now opt to fill missing values with zeros to maintain detection accuracy.\n\nKey Changes:\n1. Enhanced Missing Data Handling:\n\nChanged to ThresholdedRandomCutForest.process(double[] inputPoint, long timestamp, int[] missingValues) to support missing data in both real-time and historical analyses. The preview mode remains unchanged for efficiency, utilizing existing linear imputation techniques. (See classes: ADColdStart, ModelColdStart, ModelManager, ADBatchTaskRunner).\n\n2. Refactoring Imputation & Processing:\n\nRefactored the imputation process, failure handling, statistics collection, and result saving in Inferencer.\n\n3. Improved Imputed Value Reconstruction:\n\nReconstructed imputed values using existing mean and standard deviation, ensuring they are accurately stored in AnomalyResult. Added a featureImputed boolean tag to flag imputed values. (See class: AnomalyResult).\n\n4. Broadcast Support for HC Detectors:\n\nAdded a broadcast mechanism for HC detectors to identify entity models that haven’t received data in a given interval. This ensures models in memory process all relevant data before imputation begins. Single stream detectors handle this within existing transport messages. (See classes: ADHCImputeTransportAction, ADResultProcessor, ResultProcessor).\n\n5. Introduction of ActionListenerExecutor:\n\nAdded ActionListenerExecutor to wrap response and failure handlers in an ActionListener, executing them asynchronously using the provided ExecutorService. This allows us to handle responses in the AD thread pool.\n\nTesting:\nComprehensive testing was conducted, including both integration and unit tests. Of the 7135 lines added and 1683 lines removed, 4926 additions and 749 deletions are in tests, ensuring robust coverage.\n\nSigned-off-by: Kaituo Li \n\n* rebase from main\n\nSigned-off-by: Kaituo Li \n\n* add comment and remove redundant code\n\nSigned-off-by: Kaituo Li \n\n---------\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Add Support for Handling Missing Data in Anomaly Detection (opensearc…"}},{"before":"f3bdf1bf2d519a8de4c0f3aff04a53d9ec2f2bd3","after":"7814c6d6353f7834e1e3487c9ade1498aaf300dc","ref":"refs/heads/missing2","pushedAt":"2024-08-16T20:01:19.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"add comment and remove redundant code\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"add comment and remove redundant code"}},{"before":"835600506a6f279c42d604d974c5567612920393","after":"f3bdf1bf2d519a8de4c0f3aff04a53d9ec2f2bd3","ref":"refs/heads/missing2","pushedAt":"2024-08-14T15:23:25.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"rebase from main\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"rebase from main"}},{"before":"ff9140eb080785d4f5a92f66a16896027646c0c0","after":"835600506a6f279c42d604d974c5567612920393","ref":"refs/heads/missing2","pushedAt":"2024-08-14T00:53:12.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"rebase from main\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"rebase from main"}},{"before":"ddcd8c9d3d25a0e3878e3e4ca0670858298b4c17","after":"ff9140eb080785d4f5a92f66a16896027646c0c0","ref":"refs/heads/missing2","pushedAt":"2024-08-09T02:23:19.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Add Support for Handling Missing Data in Anomaly Detection\n\nThis PR introduces enhanced handling of missing data, giving customers the flexibility to choose how to address gaps in their data. Options include ignoring missing data (default behavior), filling with fixed values (customer-specified), zeros, or previous values. These options can improve recall in anomaly detection scenarios. For example, in this forum discussion https://forum.opensearch.org/t/do-missing-buckets-ruin-anomaly-detection/16535, customers can now opt to fill missing values with zeros to maintain detection accuracy.\n\nKey Changes:\n1. Enhanced Missing Data Handling:\n\nChanged to ThresholdedRandomCutForest.process(double[] inputPoint, long timestamp, int[] missingValues) to support missing data in both real-time and historical analyses. The preview mode remains unchanged for efficiency, utilizing existing linear imputation techniques. (See classes: ADColdStart, ModelColdStart, ModelManager, ADBatchTaskRunner).\n\n2. Refactoring Imputation & Processing:\n\nRefactored the imputation process, failure handling, statistics collection, and result saving in Inferencer.\n\n3. Improved Imputed Value Reconstruction:\n\nReconstructed imputed values using existing mean and standard deviation, ensuring they are accurately stored in AnomalyResult. Added a featureImputed boolean tag to flag imputed values. (See class: AnomalyResult).\n\n4. Broadcast Support for HC Detectors:\n\nAdded a broadcast mechanism for HC detectors to identify entity models that haven’t received data in a given interval. This ensures models in memory process all relevant data before imputation begins. Single stream detectors handle this within existing transport messages. (See classes: ADHCImputeTransportAction, ADResultProcessor, ResultProcessor).\n\n5. Introduction of ActionListenerExecutor:\n\nAdded ActionListenerExecutor to wrap response and failure handlers in an ActionListener, executing them asynchronously using the provided ExecutorService. This allows us to handle responses in the AD thread pool.\n\nTesting:\nComprehensive testing was conducted, including both integration and unit tests. Of the 7135 lines added and 1683 lines removed, 4926 additions and 749 deletions are in tests, ensuring robust coverage.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Add Support for Handling Missing Data in Anomaly Detection"}},{"before":"f56a69a716a2ea8d3b0d7cdf777f0649c95051b5","after":"ddcd8c9d3d25a0e3878e3e4ca0670858298b4c17","ref":"refs/heads/missing2","pushedAt":"2024-08-08T23:25:45.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Add Support for Handling Missing Data in Anomaly Detection\n\nThis PR introduces enhanced handling of missing data, giving customers the flexibility to choose how to address gaps in their data. Options include ignoring missing data (default behavior), filling with fixed values (customer-specified), zeros, or previous values. These options can improve recall in anomaly detection scenarios. For example, in this forum discussion https://forum.opensearch.org/t/do-missing-buckets-ruin-anomaly-detection/16535, customers can now opt to fill missing values with zeros to maintain detection accuracy.\n\nKey Changes:\n1. Enhanced Missing Data Handling:\n\nChanged to ThresholdedRandomCutForest.process(double[] inputPoint, long timestamp, int[] missingValues) to support missing data in both real-time and historical analyses. The preview mode remains unchanged for efficiency, utilizing existing linear imputation techniques. (See classes: ADColdStart, ModelColdStart, ModelManager, ADBatchTaskRunner).\n\n2. Refactoring Imputation & Processing:\n\nRefactored the imputation process, failure handling, statistics collection, and result saving in Inferencer.\n\n3. Improved Imputed Value Reconstruction:\n\nReconstructed imputed values using existing mean and standard deviation, ensuring they are accurately stored in AnomalyResult. Added a featureImputed boolean tag to flag imputed values. (See class: AnomalyResult).\n\n4. Broadcast Support for HC Detectors:\n\nAdded a broadcast mechanism for HC detectors to identify entity models that haven’t received data in a given interval. This ensures models in memory process all relevant data before imputation begins. Single stream detectors handle this within existing transport messages. (See classes: ADHCImputeTransportAction, ADResultProcessor, ResultProcessor).\n\n5. Introduction of ActionListenerExecutor:\n\nAdded ActionListenerExecutor to wrap response and failure handlers in an ActionListener, executing them asynchronously using the provided ExecutorService. This allows us to handle responses in the AD thread pool.\n\nTesting:\nComprehensive testing was conducted, including both integration and unit tests. Of the 7135 lines added and 1683 lines removed, 4926 additions and 749 deletions are in tests, ensuring robust coverage.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Add Support for Handling Missing Data in Anomaly Detection"}},{"before":null,"after":"f56a69a716a2ea8d3b0d7cdf777f0649c95051b5","ref":"refs/heads/missing2","pushedAt":"2024-08-08T23:10:51.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Add Support for Handling Missing Data in Anomaly Detection\n\nThis PR introduces enhanced handling of missing data, giving customers the flexibility to choose how to address gaps in their data. Options include ignoring missing data (default behavior), filling with fixed values (customer-specified), zeros, or previous values. These options can improve recall in anomaly detection scenarios. For example, in this forum discussion https://forum.opensearch.org/t/do-missing-buckets-ruin-anomaly-detection/16535, customers can now opt to fill missing values with zeros to maintain detection accuracy.\n\nKey Changes:\n1. Enhanced Missing Data Handling:\n\nChanged to ThresholdedRandomCutForest.process(double[] inputPoint, long timestamp, int[] missingValues) to support missing data in both real-time and historical analyses. The preview mode remains unchanged for efficiency, utilizing existing linear imputation techniques. (See classes: ADColdStart, ModelColdStart, ModelManager, ADBatchTaskRunner).\n\n2. Refactoring Imputation & Processing:\n\nRefactored the imputation process, failure handling, statistics collection, and result saving in Inferencer.\n\n3. Improved Imputed Value Reconstruction:\n\nReconstructed imputed values using existing mean and standard deviation, ensuring they are accurately stored in AnomalyResult. Added a featureImputed boolean tag to flag imputed values. (See class: AnomalyResult).\n\n4. Broadcast Support for HC Detectors:\n\nAdded a broadcast mechanism for HC detectors to identify entity models that haven’t received data in a given interval. This ensures models in memory process all relevant data before imputation begins. Single stream detectors handle this within existing transport messages. (See classes: ADHCImputeTransportAction, ADResultProcessor, ResultProcessor).\n\n5. Introduction of ActionListenerExecutor:\n\nAdded ActionListenerExecutor to wrap response and failure handlers in an ActionListener, executing them asynchronously using the provided ExecutorService. This allows us to handle responses in the AD thread pool.\n\nTesting:\nComprehensive testing was conducted, including both integration and unit tests. Of the 7135 lines added and 1683 lines removed, 4926 additions and 749 deletions are in tests, ensuring robust coverage.\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Add Support for Handling Missing Data in Anomaly Detection"}},{"before":null,"after":"fc1cfeceb29cf3ef445a852040101a7ea9c1d48b","ref":"refs/heads/validateFeature_2.x","pushedAt":"2024-07-08T19:38:36.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"follow https://github.com/opensearch-project/k-NN/pull/1795\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"follow opensearch-project/k-NN#1795"}},{"before":"be78b8655d0c451a530dd158d99a012e953d9b43","after":"1e8be88e8860633b11e9539b0efa9b64bf0f10a2","ref":"refs/heads/bash5","pushedAt":"2024-07-02T23:19:38.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Consider feature filter in model validation\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Consider feature filter in model validation"}},{"before":"df241c441ebad8cde1fd480b338184c876ff9f3a","after":"be78b8655d0c451a530dd158d99a012e953d9b43","ref":"refs/heads/bash5","pushedAt":"2024-07-02T22:16:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Consider feature filter in model validation\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Consider feature filter in model validation"}},{"before":"059d468a94b4f8d5ef6dd7c1f9a5ccc161da9444","after":"df241c441ebad8cde1fd480b338184c876ff9f3a","ref":"refs/heads/bash5","pushedAt":"2024-07-02T22:09:36.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Consider feature filter in model validation\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Consider feature filter in model validation"}},{"before":null,"after":"059d468a94b4f8d5ef6dd7c1f9a5ccc161da9444","ref":"refs/heads/bash5","pushedAt":"2024-07-02T21:27:14.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Consider feature filter in model validation\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Consider feature filter in model validation"}},{"before":null,"after":"b45df5e24afc4f204b4bdbf5c6c2b58d1a4fc697","ref":"refs/heads/bwc_2x_3","pushedAt":"2024-06-26T20:23:36.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"kaituo","name":"Kaituo Li","path":"/kaituo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5303417?s=80&v=4"},"commit":{"message":"Update BWC Test Version and Enhance Code Coverage (#1253)\n\n* Update BWC Test Version and Enhance Code Coverage\n\nThis PR updates the BWC test version to 2.16 in the build.gradle file and includes additional integration and unit tests to improve code coverage.\n\nTesting Performed:\n* Executed gradle build to ensure successful build and integration.\n\nSigned-off-by: Kaituo Li \n\n* address comments\n\nSigned-off-by: Kaituo Li \n\n---------\n\nSigned-off-by: Kaituo Li ","shortMessageHtmlLink":"Update BWC Test Version and Enhance Code Coverage (opensearch-project…"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xOFQxODoxNzo0MC4wMDAwMDBazwAAAAS6B2fa","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xOFQxODoxNzo0MC4wMDAwMDBazwAAAAS6B2fa","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wNi0yNlQyMDoyMzozNi4wMDAwMDBazwAAAARwJBuM"}},"title":"Activity · kaituo/anomaly-detection-1"}