Merge pull request #1536 from HXSecurity/develop

v1.12.0

.github/deploy/deploy-dongtai-server-test.yml

@@ -337,7 +337,7 @@ spec:
               mountPath: /opt/dongtai/dongtai_conf/conf/config.ini
               subPath: config.ini
           env:
-            - name: DONGTAI_CONCURRENCY
+            - name: DONGTAI_CONCURRENCYW
               value: -P gevent --concurrency=10
           resources:
             limits:

.github/workflows/teststate.yml

@@ -296,7 +296,7 @@ jobs:
           cp dongtai_conf/conf/config.ini.test dongtai_conf/conf/config.ini
           mkdir -p /tmp/logstash/report/{img,word,pdf,excel} && mkdir -p /tmp/iast_cache/package && mkdir -p /tmp/logstash/batchagent 
           python3 ./deploy/docker/version_update.py || true
-          pip install Cython==3.0.0a11
+          pip install Cython==3.0.0b3
           python setup.py build_ext --inplace
           find . -name "*.so"  | grep test | xargs rm
           coverage run --source='.' manage.py test
@@ -351,7 +351,8 @@ jobs:
           export DOC=TRUE
           mkdir -p /tmp/logstash/report/{img,word,pdf,excel} && mkdir -p /tmp/iast_cache/package && mkdir -p /tmp/logstash/batchagent 
           python3 ./deploy/docker/version_update.py || true
-          pip install Cython==3.0.0a11
+          pip install Cython==3.0.0b3
+          python setup.py build_ext --inplace
           python setup.py build_ext --inplace
           python3 manage.py updatedepartmenttoken
           python3 manage.py runserver 0.0.0.0:8000 > webapi.log &

.gitignore

@@ -32,3 +32,4 @@ celerybeat.pid
 *.mo
 *.o
 *.c
+*.prof

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.10-slim
+FROM python:3.10-slim-bullseye
 ARG VERSION
 ENV DEBIAN_FRONTEND=noninteractive
 ENV LANG=en_US.UTF-8
@@ -7,10 +7,10 @@ ENV LANGUAGE=en_US.UTF-8
 ENV TZ=Asia/Shanghai
 
 RUN apt-get update -y \
-    && apt install -y gettext gcc make cmake libmariadb-dev curl libc6-dev unzip cron \
+    && apt install -y gettext gcc make cmake libmariadb-dev curl libc6-dev libxrender1 libxtst6 libxi6 unzip cron \
     fonts-wqy-microhei vim build-essential ninja-build cython3 pybind11-dev libre2-dev locales \
 #   htop sysstat net-tools iproute2 procps lsof \
-    openjdk-11-jdk \
+    zip libjpeg62 \
     && sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen \
     && ALIMARCH=`arch` && curl -L https://charts.dongtai.io/apk/${ALIMARCH}/wkhtmltopdf -o /usr/bin/wkhtmltopdf \
     && chmod +x /usr/bin/wkhtmltopdf

Pipfile

@@ -24,12 +24,12 @@ idna = "==2.10"
 lxml = "==4.9.1"
 mysqlclient = "*"
 python-docx = "==0.8.11"
-requests = "==2.25.1"
+requests = "==2.31.0"
 six = "==1.15.0"
 urllib3 = "==1.26.5"
 xlwt = "==1.3.0"
 pyre2 = "~=0.3.6"
-celery = "==5.2.7"
+celery = "==5.3.0rc1"
 redis = "==4.4.4"
 openpyxl = "==3.0.9"
 id-validator = "==1.0.20"
@@ -50,8 +50,8 @@ packaging = "==21.3"
 docxtpl = "==0.16.0"
 docxcompose = "==1.3.4"
 django-elasticsearch-dsl = "==7.2.2"
-asyncio-gevent = "==0.2.1"
-gevent = "==21.12.0"
+#asyncio-gevent = "==0.2.1"
+gevent = "==22.10.2"
 ddt = "==1.6.0"
 boto3 = "==1.24.59"
 mypy = "==1.0.1"
@@ -69,14 +69,21 @@ botocore = "==1.27.91"
 setuptools = "==65.5.1"
 elasticsearch = "==7.17.7"
 django-mock-queries = "==v2.1.7"
-cryptography = "==39.0.1"
+cryptography = "==41.0.0"
 jsonschema = "==4.17.0"
 pillow = "==9.3.0"
 pyrsistent = "==0.19.1"
 pytz = "==2022.6"
 types-pyyaml = ">=6.0.12.2"
 uwsgi = "==2.0.21"
 marisa-trie = "==0.8.0"
+gunicorn = "==20.1.0"
+celery-singleton = "*"
+djangorestframework-dataclasses = "*"
+django-seriously = "*"
+dataclasses-json = "*"
+django-silk = "*"
+types-python-dateutil = "*"
 
 [dev-packages]
 

README-zh.md

@@ -15,7 +15,7 @@
 
 ## DongTai是什么?
 
-DongTai是一款开源的被动式交互式安全测试(IAST)产品，通过动态hook和污点跟踪算法等实现**通用漏洞检测**、**多请求关联漏洞检测(包括但不限于越权漏洞、未授权访问)**、**第三方组件漏洞检测**等，目前支持Java、Python两种语言的应用漏洞检测。
+洞态IAST是一款开源的交互式安全测试(IAST)产品，可通过被动插桩模式实现JAVA应用的通用漏洞及第三方组件漏洞的实时检测，非常适合在开发流水线的测试阶段使用。
 
 ## 项目结构
 
@@ -34,12 +34,12 @@ DongTai是一款开源的被动式交互式安全测试(IAST)产品，通过动
 
 ## 技术架构
 
-"火线-洞态IAST"具有多个基础服务，包括：`DongTai-web`、`DongTai-webapi`、`DongTai-openapi`、`DongTai-engine`、`agent`、`DongTai-Base-Image`、`DongTai-Plugin-IDEA`，其中：
+"火线-洞态IAST"具有多个基础服务，包括：`DongTai-web`、`DongTai`、 `agent`、`DongTai-Base-Image`、`DongTai-Plugin-IDEA`，其中：
 
 - `DongTai-web`是DongTai的产品页面，用于处理用户与洞态的交互
-- `DongTai-webapi`负责处理用户的相关操作
-- `DongTai-openapi`用于处理`agent`上报的注册/心跳/调用方法/第三方组件/错误日志等数据，下发hook策略，下发探针控制指令等
-- `DongTai-engine`根据调用方法数据和污点跟踪算法分析HTTP/HTTPS/RPC请求中是否存在漏洞，同时负责其它相关的定时任务
+- `DongTai>>dongtai_web`负责处理用户的相关操作的API
+- `DongTai>>dongtai_protocol`用于处理`agent`上报的注册/心跳/调用方法/第三方组件/错误日志等数据，下发hook策略，下发探针控制指令等
+- `DongTai>>dongtai_engine` 根据调用方法数据和污点跟踪算法分析HTTP/HTTPS/RPC请求中是否存在漏洞，同时负责其它相关的定时任务
 - `agent`是DongTai的探针模块，包含不同编程语言的数据采集端，用于采集应用运行时的数据并上报至`DongTai-OpenAPI`服务
 - `DongTai-Base-Image`包含洞态运行时依赖的基础服务，包括：MySql、Redis
 - `DongTai-Plugin-IDEA`是Java探针对应的IDEA插件，可通过插件直接运行Java探针，直接在IDEA中检测漏洞

README.md

@@ -15,7 +15,7 @@
 
 ## About DongTai IAST
 
-`DongTai IAST` is an open-source passive interactive security testing (IAST) product. It uses dynamic hooks and taint tracking algorithms to achieve **universal vulnerability detection** and **multiples request associated with vulnerability detection (including but not limited to unauthorized vulnerabilities, overpower vulnerabilities)**, **Third-party component vulnerability detection**, etc. Currently, applications in Java and Python are supported for vulnerability detection.
+`Dongtai IAST` is an open-source Interactive Application Security Testing (IAST) tool that enables real-time detection of common vulnerabilities in Java applications and third-party components through passive instrumentation. It is particularly suitable for use in the testing phase of the development pipeline.
 
 
 ## Project structure
@@ -34,14 +34,14 @@
 
 ## Architecture
 
-`DongTai IAST` has multiple basic services, including `DongTai-web`, `DongTai-webapi`, `DongTai-openapi`, `DongTai-engine`, `agent`, `DongTai-deploy`, `DongTai-Base-Image` and `DongTai-Plugin-IDEA`:
+`DongTai IAST` has multiple basic services, including `DongTai-web`, `DongTai`， `agent`, `DongTai-Base-Image` and `DongTai-Plugin-IDEA`:
 
 - `DongTai-web` is the product page of DongTai, which is used to handle the interaction between users and cave states.
-- `DongTai-webapi` is responsible for handling user-related operations.
-- `DongTai-openapi` is used to process the registration/heartbeat/call method/third-party component/error log data reported by `agent`, issue hook strategy, issue probe control commands, etc.
-- `DongTai-engine` analyzes whether there are vulnerabilities in HTTP/HTTPS/RPC requests according to the calling method data and taint tracking algorithm, and is also responsible for other related timing tasks.
+- `DongTai>>dongtai_web` is responsible for handling user-related operations.
+- `DongTai>>dongtai_protocol` is used to process the registration/heartbeat/call method/third-party component/error log data reported by `agent`, issue hook strategy, issue probe control commands, etc.
+- `DongTai>>dongtai_engine` analyzes whether there are vulnerabilities in HTTP/HTTPS/RPC requests according to the calling method data and taint tracking algorithm, and is also responsible for other related timing tasks.
 - `agent` is a probe module of DongTai, including data collection terminals in different programming languages, used to collect data during application runtime and report to the `DongTai-OpenAPI` service.
-- `DongTai-deploy` is used for the deployment of DongTai IAST, including docker-compose single-node deployment, Kubernetes cluster deployment, etc. If you want a deployment plan, you can add features or contribute to the deployment plan.
+- `DongTai>>deploy` is used for the deployment of DongTai IAST, including docker-compose single-node deployment, Kubernetes cluster deployment, etc. If you want a deployment plan, you can add features or contribute to the deployment plan.
 - `DongTai-Base-Image` contains the basic services that DongTai depends on runtime, including MySql, Redis.
 - `DongTai-Plugin-IDEA` is the IDEA plug-in corresponding to the Java probe. You can run the Java probe directly through the plug-in and detect the vulnerabilities directly in IDEA.
 

deploy/docker/entrypoint.sh

@@ -15,7 +15,7 @@ elif [ "$1" = "worker-high-freq" ]; then
 elif [ "$1" = "worker-es" ]; then
 	celery -A dongtai_conf worker -l info -Q dongtai-es-save-task $DONGTAI_CONCURRENCY -E --pidfile=  
 elif [ "$1" = "worker-sca" ]; then
-	celery -A dongtai_conf worker -l info -Q dongtai-sca-task $DONGTAI_CONCURRENCY -E --pidfile=  
+	celery -A dongtai_conf worker -l info -Q dongtai-sca-task,dongtai-api-route-handler $DONGTAI_CONCURRENCY -E --pidfile=  
 elif [ "$1" = "worker-other" ]; then
 	celery -A dongtai_conf worker -l info -X dongtai-periodic-task,dongtai-method-pool-scan,dongtai-replay-vul-scan,dongtai-sca-task $DONGTAI_CONCURRENCY -E --pidfile= 
 elif [ "$1" = "beat" ]; then

dongtai_common/common/utils/__init__.py

@@ -47,9 +47,12 @@ def make_hash(obj):
     return hash(tuple(frozenset(new_obj.items())))
 
 
-def cached(function,
-           random_range: tuple = (50, 100),
-           use_celery_update: bool = False):
+def cached(
+    function,
+    random_range: tuple = (50, 100),
+    use_celery_update: bool = False,
+    cache_logic_none: bool = True,
+):
     """Return a version of this function that caches its results for
     the time specified.
 
@@ -70,17 +73,23 @@ def get_cache_or_call(*args, **kwargs):
         # the cache
         cache_key = make_hash(
             (function.__module__ + function.__name__, args, kwargs))
-        cached_result = cache.get(cache_key)
+        #cache_key = function.__module__ + function.__name__
+        cached_result = cache.get(cache_key, "Not such key")
         if random_range:
             cache_time = random.randint(*random_range)
         if use_celery_update:
             function_flush.apply_async(args=(function.__module__,
                                              function.__name__, cache_time,
                                              tuple(args), kwargs))
-        if cached_result is None:
+        if cached_result == "Not such key":
             result = function(*args, **kwargs)
-            cache.set(cache_key, result, cache_time)
+            if cache_logic_none and result is None:
+                cache.set(cache_key, result, cache_time)
+            else:
+                cache.set(cache_key, result, cache_time)
             return result
+        elif cached_result is None:
+            return cached_result
         else:
             return cached_result
 
@@ -89,32 +98,50 @@ def get_cache_or_call(*args, **kwargs):
     return get_cache_or_call
 
 
-def cached_decorator(random_range, use_celery_update=False):
+def disable_cache(function, args=(), kwargs={}):
+    cache_key = make_hash(
+        (function.__module__ + function.__name__, args, kwargs))
+    cache.delete(cache_key)
+
+
+def cached_decorator(random_range,
+                     use_celery_update=False,
+                     cache_logic_none=True):
 
     def _noname(function):
-        return cached(function,
-                      random_range,
-                      use_celery_update=use_celery_update)
+        return cached(
+            function,
+            random_range,
+            use_celery_update=use_celery_update,
+            cache_logic_none=cache_logic_none,
+        )
 
     return _noname
 
 
+@cached_decorator(random_range=(60, 120), use_celery_update=False)
+def get_user_from_department_key(key):
+    from dongtai_common.models.department import Department
+    from dongtai_common.models.user import User
+    from rest_framework import exceptions
+    department = Department.objects.get(token=key)
+    principal = User.objects.filter(pk=department.principal_id).first()
+    user = principal if principal else User.objects.filter(pk=1).first()
+    user.using_department = department
+    return user
+
 class DepartmentTokenAuthentication(TokenAuthentication):
 
     keyword = 'Token GROUP'
     model = None
 
-    def authenticate_credentials(self, key):
+    def auth_decodedenticate_credentials(self, key):
         from dongtai_common.models.department import Department
         from dongtai_common.models.user import User
         from rest_framework import exceptions
-        model = Department
         try:
-            department = model.objects.get(token=key)
-            principal = User.objects.filter(pk=department.principal_id).first()
-            user = principal if principal else User.objects.filter(pk=1).first()
-            user.using_department = department
-        except model.DoesNotExist:
+            user = get_user_from_department_key(key)
+        except Department.DoesNotExist:
             raise exceptions.AuthenticationFailed(_('Invalid token.'))
         return (user, key)
 
@@ -125,4 +152,4 @@ def authenticate(self, request):
             return None
         token = auth.lower().replace(self.keyword.lower().encode(), b'',
                                      1).decode()
-        return self.authenticate_credentials(token)
+        return self.auth_decodedenticate_credentials(token)

dongtai_common/models/asset_vul_v2.py

@@ -0,0 +1,69 @@
+import uuid
+from dongtai_common.models.agent import IastAgent
+from django.core.cache import cache
+from django_elasticsearch_dsl.search import Search
+from dongtai_conf.settings import ASSET_VUL_INDEX
+from django_elasticsearch_dsl import Document, fields
+from django_elasticsearch_dsl.registries import registry
+from dongtai_common.models.assetv2 import AssetV2Global, AssetRiskLevel
+from django.db import models
+from dongtai_common.utils.settings import get_managed
+from dongtai_common.models.vulnerablity import IastVulnerabilityStatus
+from dongtai_common.models.vul_level import IastVulLevel
+
+
+class IastAssetVulV2(models.Model):
+    vul_name = models.CharField(max_length=255, blank=True, null=True)
+    vul_detail = models.TextField(blank=True, null=True)
+    # 漏洞类型等级
+    level = models.IntegerField(choices=AssetRiskLevel.choices,
+                                blank=True,
+                                db_column="level_id",
+                                default=AssetRiskLevel.LOW)
+    update_time = models.IntegerField(blank=True, null=True)
+    create_time = models.IntegerField(blank=True, null=True)
+    references = models.JSONField(blank=True, null=True, default=list)
+    change_time = models.IntegerField(blank=True, null=True)
+    published_time = models.IntegerField(blank=True, null=True)
+    vul_id = models.CharField(max_length=255,
+                              blank=True,
+                              null=True,
+                              unique=True)
+    vul_type = models.JSONField(blank=True, null=True)
+    vul_codes = models.JSONField(blank=True, null=True)
+    affected_versions = models.JSONField(blank=True, null=True)
+    unaffected_versions = models.JSONField(blank=True, null=True)
+
+    class Meta:
+        managed = True
+        db_table = 'iast_asset_vul_v2'
+
+
+class IastVulAssetRelationV2(models.Model):
+    asset_vul = models.ForeignKey(IastAssetVulV2,
+                                  on_delete=models.DO_NOTHING,
+                                  db_constraint=False,
+                                  db_column='vul_id',
+                                  to_field="vul_id")
+    asset = models.ForeignKey(AssetV2Global,
+                              on_delete=models.DO_NOTHING,
+                              db_constraint=False,
+                              db_column='asset',
+                              to_field="aql")
+
+    class Meta:
+        managed = get_managed()
+        db_table = 'iast_asset_vul_v2_relation'
+
+
+#class IastPackageGAInfo(models.Model):
+#    package_name = models.ForeignKey(AssetV2Global,
+#                                     on_delete=models.DO_NOTHING,
+#                                     db_constraint=False,
+#                                     db_column='package_name')
+#    affected_versions = models.JSONField(blank=True, null=True, default=list)
+#    unaffected_versions = models.JSONField(blank=True, null=True, default=list)
+#
+#    class Meta:
+#        managed = get_managed()
+#        db_table = 'iast_asset_v2_ga_info'