forked from intel-analytics/BigDL-2.x
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* add anomaly-detection-hd * modified according to comments * Data download sccipt added * visualization optimized * minor fix * typo correction * refactor autoencoder
- Loading branch information
0 parents
commit 11c0078
Showing
2 changed files
with
785 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# Anomaly Detection | ||
This is a simple example of unsupervised anomaly detection using Analytics Zoo Keras-Style API. In statistics, anomalies, also known as outliers, are observation points that are distant from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error, and it can cause serious problems in statistical analyses. | ||
We use an unsupervised neural network to perform a mapping from the original space to another space (encoder) and a mapping back from this new space to the original one (decoder). | ||
Since outliers are rare and different, that the auto-encoder will not learn to map those objects correctly, inducing a higher reconstruction error. This reconstruction error will result in a higher outlier score. | ||
(Retrieved from https://edouardfouche.com/Neural-based-Outlier-Discovery/) | ||
|
||
## Environment | ||
* Python 2.7 | ||
* Apache Spark 2.2.0 (This version needs to be same with the version you use to build Analytics Zoo) | ||
|
||
## Run with Jupyter | ||
* Download Analytics Zoo and build it. | ||
* Run `export SPARK_HOME=the root directory of Spark`. | ||
* Run `export ANALYTICS_ZOO_HOME=the dist directory under the Analytics Zoo project`. | ||
* Run `$ANALYTICS_ZOO_HOME/bin/data/HiCS/get_HiCS.sh` to download the dataset. (More data can be found from [here](https://www.ipd.kit.edu/~muellere/HiCS/)). | ||
* Run the following bash command to start the jupyter notebook. Change parameter settings as you need, ie `MASTER = local[physcial_core_number]`. | ||
```bash | ||
MASTER=local[*] | ||
${ANALYTICS_ZOO_HOME}/bin/jupyter-with-zoo.sh \ | ||
--master ${MASTER} \ | ||
--driver-cores 4 \ | ||
--driver-memory 32g \ | ||
--total-executor-cores 4 \ | ||
--executor-cores 4 \ | ||
--executor-memory 32g | ||
``` |
Oops, something went wrong.