diff --git a/README-ZH.md b/README-ZH.md index 9f5b760a0..33f0cf8b8 100644 --- a/README-ZH.md +++ b/README-ZH.md @@ -35,15 +35,16 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com        支持流式作业的多版本管理、全生命周期管理、监控告警、checkpoint 和 savepoint 管理能力。 -![流式应用生产](docs/images/生产中心.png) +![流式应用生产](docs/images/stream_product_center.png)        流式应用运行情况: -![流式应用运行情况](docs/images/流式作业运行情况.png) +![流式应用运行情况](docs/images/stream_job_detail.png)        流式应用参数配置: -![流式应用配置](docs/images/流式应用配置.png) +![流式应用配置](docs/images/stream_job_config_1.png) +![流式应用配置](docs/images/stream_job_config_2.png)        更多功能,请参考:[Streamis 用户手册](docs/zh_CN/0.2.0/使用文档/Streamis用户手册.md)。 @@ -51,10 +52,10 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com ## 依赖的生态组件 -| 依赖的应用工具 | 描述 | Streamis 兼容版本 | -| -------------- | -------------------------------------------------------------- | --------------| -| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | 数据应用开发管理集成框架。以工作流式的图形化拖拽开发体验,将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出应用等,数据应用开发全流程场景需求。 | >= DSS1.0.1(已发布)| -| [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis,通过提供 REST/WebSocket/JDBC/SDK 等标准接口,上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | >= Linkis1.0.3(已发布) | +| 依赖的应用工具 | 描述 | Streamis 兼容版本 | +|--------------|---------------------------------------------------------------|--------------| +| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | 数据应用开发管理集成框架。以工作流式的图形化拖拽开发体验,将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出应用等,数据应用开发全流程场景需求。 | >= DSS1.1.0(已发布)| +| [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis,通过提供 REST/WebSocket/JDBC/SDK 等标准接口,上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | >= Linkis1.1.1(已发布) | ---- @@ -72,13 +73,7 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com ## 编译和安装部署 -       请参照 [编译指引](docs/zh_CN/0.2.0/development/Streamis编译文档.md) ,用于编译 Streamis。 - -       在安装Streamis之前,请先安装 Linkis1.0.3 和 DSS1.0.1,可参照:[DSS + Linkis 安装部署文档](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md) - -       同时,请确保 Linkis Flink 引擎已可正常使用,可参照:[Linkis Flink 引擎插件安装文档](https://github.com/WeBankFinTech/Linkis-Doc/blob/master/zh_CN/Deployment_Documents/EngineConnPlugin%E5%BC%95%E6%93%8E%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3.md) - -       最后,请参考 [Streamis 安装部署文档](docs/zh_CN/0.2.0/Streamis安装文档.md) ,用于安装部署 Streamis。 +       请参考 [Streamis 安装部署文档](docs/zh_CN/0.2.0/Streamis安装文档.md) ,用于安装部署 Streamis 及其依赖环境。 ---- @@ -100,8 +95,10 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com | | 支持 FlinkSQL 和 FlinkJar 包等方式发布 | 支持 | | | 流式应用的多版本管理能力 | 支持 | | | 流式应用的参数配置和告警能力 | 支持 | - | 服务高可用 | 应用高可用,服务多点,故障不影响使用 | 支持 | + | 服务高可用 | 应用高可用,服务多点,状态快照实现容错处理,故障不影响使用 | 支持 | | 系统管理 | 节点、资源管理 | 支持 | + |权限管理 |任务的操作权限控制 |支持 | + ---- ## 架构 diff --git a/README.md b/README.md index 262293b2c..76c3121df 100644 --- a/README.md +++ b/README.md @@ -53,8 +53,8 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/ | Depended Component | Description | Streamis compatibility | | -------------- | -------------------------------------------------------------- | --------------| -| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. | >= DSS1.0.1 (Released) | -| [Linkis](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | >= Linkis1.0.3 (Released) | +| [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. | >= DSS1.1.0 (Released) | +| [Linkis](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | >= Linkis1.1.1 (Released) | ## Demo Trial environment @@ -70,13 +70,7 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/ ## Compile and install deployment -       Please refer to [Compilation guidelines](docs/en_US/0.1.0/StreamisCompiledDocument.md) used to compile Streamis. - -       Before installing Streamis, please install Linkis1.0.3 and DSS1.0.1 first, please refer to: [DSS deployment documents](https://github.com/WeBankFinTech/DataSphereStudio/blob/master/docs/en_US/ch1/DataSphereStudio_Compile_Manual.md) and [Linkis deployment documents](https://github.com/WeBankFinTech/Linkis-Doc/blob/master/en_US/Deployment_Documents/Quick_Deploy_Linkis1.0.md) - -       At the same time, please make sure that the Linkis Flink engine can be used normally, please refer to: [Linkis Flink Engine Plug-in Installation Document](https://github.com/WeBankFinTech/Linkis-Doc/blob/master/en_US/Deployment_Documents/EngineConnPlugin_installation_document.md) - -       Finally, please refer to [Streamis Installation and Deployment Document](docs/en_US/0.1.0/StreamisDeployment.md) for installing and deploying Streamis. +please refer to [Streamis Installation and Deployment Document](docs/en_US/0.2.0/StreamisDeployment.md) for installing and deploying Streamis. ---- @@ -98,8 +92,9 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/ | | Support FlinkSQL and FlinkJar package release | Support | | | Multi-version management capabilities | Support | | | Configuration and alert management capabilities | Support | -| Service high availability | Multiple services, failure does not affect the use | Application high availability | +| Service high availability | Multiple services,State snapshot for fault tolerance, failure does not affect the use | Application high availability | | System Management | Node and Resource Management | Support | +| Permission management | Task operation permission control | Support | ---- diff --git a/docs/en_US/0.1.0/StreamisDeployment.md b/docs/en_US/0.1.0/StreamisDeployment.md deleted file mode 100644 index e676ede3e..000000000 --- a/docs/en_US/0.1.0/StreamisDeployment.md +++ /dev/null @@ -1,165 +0,0 @@ -# Streamis installation and deployment documentation - -## 1. Component introduction ----------- - -Streamis0.1.0 provides the Streamis-JobManager component, the role of the component is
-1. Publish streaming applications
-2. Set streaming application parameters, such as the number of Flink slots, checkpoint related parameters, etc.
-3. Manage streaming applications (e.g. start and stop)
-4. Streaming application monitoring
- - -## 2. Code compilation ----------- -If you have already obtained the installation package, you can skip this step
-The background compilation method is as follows -``` -cd ${STREAMIS_CODE_HOME} -mvn -N install -mvn clean install -``` -The front-end compilation method is as follows -```bash -cd ${STREAMIS_CODE_HOME}/web -npm i -npm run build -``` - - -## 3. Installation preparation -### 3.1 Basic environment installation -        The following software must be installed: - --MySQL (5.5+), [How to install MySQL](https://www.runoob.com/mysql/mysql-install.html) --JDK (above 1.8.0_141), [How to install JDK](https://www.runoob.com/java/java-environment-setup.html) - -### 3.2Linkis environment -The execution of Streamis depends on Linkis, and it needs to be version 1.0.3 and above, so you need to install Linkis above 1.0.3 and ensure that the Flink engine can be used normally. Specifically, you can create a new and edit a flinksql script on Scriptis -And execute. If flinksql can be executed correctly, it means that the linkis1.0.3 environment is normal. - -### 3.3 Package Preparation -Obtain the software package from the third step and upload it to the installation directory of the server, such as /appcom/Install/streamis -```bash -cd /appcom/Install/streamis -tar -xvf wedatasphere-streamis-${streamis-version}-dist.tar.gz -``` - -### 3.4 Modify the database configuration -```bash -vi conf/db.sh -#Configure basic database information -``` - -### 3.5 Modify the basic configuration file - -```bash -vi conf/config.sh -#Configure service port information -#Configure Linkis service information -``` -## 4. Installation and startup ----------- - --Background installation -```bash -sh bin/install.sh -``` - --The install.sh script will ask you if you need to initialize the database and import metadata. - -     Because the user is worried that the user repeatedly executes the install.sh script to clear the user data in the database, when the install.sh is executed, the user will be asked if they need to initialize the database and import metadata. - -     **Yes must be selected for the first installation**. - - -- start up -```bash -sh bin/start-streamis.sh -``` - --Start verification -Verification method, because Streamis and Linkis use a set of Eureka, you need to check whether the Eureka page of Linkis already contains Streamis services, as shown in the figure, -![components](../../images/zh_CN/eureka_streamis.png) - - - --Front-end deployment - -1. Install nginx - -```bash -sudo yum install -y nginx -``` -2. Deploy the front-end package -``` -mkdir ${STREAMIS_FRONT_PATH} -cd ${STREAMIS_FRONT_PATH} -#Place the front-end package -unzip streamis-web.zip -``` -3. Modify the nginx configuration file
- -```bash -cd /etc/nginx/conf.d -vi streamis.conf -# Copy the following template and modify it according to the actual situation -``` -``` -server { - listen 9088;# access port - server_name localhost; - location / { - root ${STREAMIS_FRONT_PAH}; # Please modify it to the appropriate static file directory of Streamis - index index.html index.html; - } - location /api { - proxy_pass http://${Linkis_GATEWAY_IP}:${LINKIS_GATEWY_PORT}; #Back-end Linkis address, please modify it to the ip and port of the Linkis gateway - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header x_real_ipP $remote_addr; - proxy_set_header remote_addr $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_http_version 1.1; - proxy_connect_timeout 4s; - proxy_read_timeout 600s; - proxy_send_timeout 12s; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection upgrade; - } - - #error_page 404 /404.html; - # redirect server error pages to the static page /50x.html - # - error_page 500 502 503 504 /50x.html; - location = /50x.html { - root /usr/share/nginx/html; - } -} -``` -4. Restart nginx -```bash -sudo systemctl restart nginx -``` - -## 5. Access to DSS -Streamis0.1.0 version access to DSS only implements the first-level specification, that is, you can jump to the Streamis front-end without password on the DSS page. Specifically, you need to configure the basic information of streamis into the DSS database, as follows: You need to replace the ip and port in the first line of sql. -```roomsql -SET @STREAMIS_INSTALL_IP_PORT='127.0.0.1:9088'; -SET @URL = replace('http://STREAMIS_IP_PORT','STREAMIS_IP_PORT', @STREAMIS_INSTALL_IP_PORT); -SET @HOMEPAGE_URL = replace('http://STREAMIS_IP_PORT','STREAMIS_IP_PORT', @STREAMIS_INSTALL_IP_PORT); -SET @PROJECT_URL = replace('http://STREAMIS_IP_PORT','STREAMIS_IP_PORT', @STREAMIS_INSTALL_IP_PORT); - - -delete from `dss_application` WHERE `name` ='STREAMIS'; -INSERT INTO `dss_application`(`name`,`url`,`is_user_need_init`,`level`,`user_init_url`,`exists_project_service`,`project_url`,`enhance_json`,`if_iframe`,`homepage_url`,`redirect_url`) VALUES ('STREAMIS', @URL, 0, 1, NULL, 0, @PROJECT_URL,'', 1, @HOMEPAGE_URL, @REDIRECT_URL); - -select @dss_STREAMIS_applicationId:=id from `dss_application` WHERE `name` ='STREAMIS'; - -select @dss_onestop_menu_id:=id from `dss_onestop_menu` where `name` ='Application Development'; - -delete from `dss_onestop_menu_application` WHERE title_en ='STREAMIS'; -INSERT INTO `dss_onestop_menu_application` (`application_id`, `onestop_menu_id`, `title_en`, `title_cn`, `desc_en`, `desc_cn`, `labels_en`, `labels_cn`, `is_active`, `access_button_en`, `access_button_cn`, `manual_button_en`, `manual_button_cn`, `manual_button_url`, `icon`, `order`, `create_by`, `create_time`, `last_update_time`, `last_update_user`, `image`) -VALUES(@dss_STREAMIS_applicationId, @dss_onestop_menu_id,'STREAMIS','Real-time computing platform','Streamis is a streaming application development management system. Based on DataSphereStudio framing capabilities and the underlying docking of Linkis Flink engine, users can complete the development, debugging, release and production management of streaming applications at low cost','Streamis is a streaming application development and management system jointly built by WeBank in cooperation with Tianyi Cloud, Xian Weng Technology and Samoyed Cloud. Based on the framework capabilities of DataSphere Studio, and the underlying docking Linkis The Flink engine allows users to complete the development, debugging, release and production management of streaming applications at low cost.','Streamis','real-time computing platform','1','Enter Streamis','enter Streamis','user manual','User Manual','http://127.0.0.1:8088/wiki/scriptis/manual/workspace_cn.html','shujujiaohuan-logo',NULL,NULL,NULL,NULL,NULL,'shujujiaohuan-icon '); -``` -After executing sql, you may need to restart dss-framework-project-server to refresh the memory. \ No newline at end of file diff --git a/docs/en_US/0.1.0/FlinkEnginePluginCompileDocument.md b/docs/en_US/0.2.0/FlinkEnginePluginCompileDocument.md similarity index 100% rename from docs/en_US/0.1.0/FlinkEnginePluginCompileDocument.md rename to docs/en_US/0.2.0/FlinkEnginePluginCompileDocument.md diff --git a/docs/en_US/0.1.0/StreamisAppConnDesignDocument.md b/docs/en_US/0.2.0/StreamisAppConnDesignDocument.md similarity index 100% rename from docs/en_US/0.1.0/StreamisAppConnDesignDocument.md rename to docs/en_US/0.2.0/StreamisAppConnDesignDocument.md diff --git a/docs/en_US/0.1.0/StreamisCompiledDocument.md b/docs/en_US/0.2.0/StreamisCompiledDocument.md similarity index 100% rename from docs/en_US/0.1.0/StreamisCompiledDocument.md rename to docs/en_US/0.2.0/StreamisCompiledDocument.md diff --git a/docs/en_US/0.2.0/StreamisDeployment.md b/docs/en_US/0.2.0/StreamisDeployment.md new file mode 100644 index 000000000..a40a2dcc4 --- /dev/null +++ b/docs/en_US/0.2.0/StreamisDeployment.md @@ -0,0 +1,165 @@ +# Streamis installation and deployment documentation + +## 1. Component introduction +Streamis0.1.0 provides the Streamis-JobManager component, the role of the component is
+1. Publish streaming applications
+2. Set streaming application parameters, such as the number of Flink slots, checkpoint related parameters, etc.
+3. Manage streaming applications (e.g. start and stop)
+4. Streaming application monitoring
+ + +## 2. Code compilation +Streamis does not require manual compilation. You can download the installation package directly for deployment. Please [click to download the installation package](https://github.com/WeBankFinTech/Streamis/releases) + +If you have already obtained the installation package, you can skip this step
+ +- The background compilation method is as follows +``` +cd ${STREAMIS_CODE_HOME} +mvn -N install +mvn clean install +``` +After successful compilation, the installation package will be generated in the 'assembly/target' directory of the project + +- The front-end compilation method is as follows + +Pre dependency: nodejs, python 2.0 + +```bash +cd ${STREAMIS_CODE_HOME}/web +npm i +npm run build +``` +After the compilation is successful, the installation package will be generated in the `${STREAMIS_CODE_HOME}/web` directory + +## 3. Installation preparation +### 3.1 Basic environment installation +        The following software must be installed: + +- MySQL (5.5+), [How to install MySQL](https://www.runoob.com/mysql/mysql-install.html) +- JDK (above 1.8.0_141), [How to install JDK](https://www.runoob.com/java/java-environment-setup.html) + +### 3.2 Linkis and DSS environments +- The execution of Streamis depends on Linkis, and it needs to be version 1.1.1 and above, so you need to install Linkis above 1.1.1 and ensure that the Flink engine can be used normally. Specifically, you can create a new and edit a flinksql script on Scriptis +- Datasphere studio (> =1.1.0), the development and debugging of streaming jobs depend on DSS scriptis, and the streaming production center needs to be embedded in the DSS engineering framework system, so it depends on * * dss-1.1.0 * * and above. + +Before the formal installation of streamis, please install linkis1.1.1 and dss1.1.0 or above, and ensure that the linkis Flink engine and DSS can be used normally. For the installation of DSS and linkis, please refer to the [dss & linkis one click installation and deployment document](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/DSS%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md). + +How to verify that DSS and linkis are basically available? You can create a flinksql script on DSS scriptis and execute it. If flinksql can execute correctly and return the result set, it means that the DSS and linkis environments are available. + + +## 4. Installation and startup + +### Background installation + +1.installation package preparation + +Upload the installation package to the installation directory of the Linux server (currently only supports linux environment deployment), such as /appcom/install/streams, and then extract it: + +```bash +cd /appcom/Install/streamis +tar -xvf wedatasphere-streamis-${streamis-version}-dist.tar.gz +``` + +2.Modify the database configuration +```bash +vi conf/db.sh +#Configure basic database information +``` + +3.Modify the basic configuration file + +```bash +vi conf/config.sh +#Configure service port information +#Configure Linkis service information +``` +4.Installation +```bash +sh bin/install.sh +``` + +- The install.sh script will ask you if you need to initialize the database and import metadata. + +     Because the user is worried that the user repeatedly executes the install.sh script to clear the user data in the database, when the install.sh is executed, the user will be asked if they need to initialize the database and import metadata. + +     **Yes must be selected for the first installation**. + +5.start up +```bash +sh bin/start.sh +``` + +- Start verification +Verification method, because Streamis and Linkis use a set of Eureka, you need to check whether the Eureka page of Linkis already contains Streamis services, as shown in the figure, +![components](../../images/zh_CN/eureka_streamis.png) + + + +### Front-end deployment + +1.Install nginx + +```bash +sudo yum install -y nginx +``` +2.Deploy the front-end package +``` +mkdir ${STREAMIS_FRONT_PATH} +cd ${STREAMIS_FRONT_PATH} +#Place the front-end package +unzip streamis-{streamis-version}.zip +``` +3.Modify the nginx configuration file
+ +```bash +cd /etc/nginx/conf.d +vi streamis.conf +# Copy the following template and modify it according to the actual situation +``` +``` +server { + listen 9088;# access port + server_name localhost; + location / { + root ${STREAMIS_FRONT_PAH}; # Please modify it to the appropriate static file directory of Streamis + index index.html index.html; + } + location /api { + proxy_pass http://${Linkis_GATEWAY_IP}:${LINKIS_GATEWY_PORT}; #Back-end Linkis address, please modify it to the ip and port of the Linkis gateway + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header x_real_ipP $remote_addr; + proxy_set_header remote_addr $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_http_version 1.1; + proxy_connect_timeout 4s; + proxy_read_timeout 600s; + proxy_send_timeout 12s; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection upgrade; + } + + #error_page 404 /404.html; + # redirect server error pages to the static page /50x.html + # + error_page 500 502 503 504 /50x.html; + location = /50x.html { + root /usr/share/nginx/html; + } +} +``` +4.Load nginx configuration +```bash +sudo nginx -s reload +``` + +## 5. Access to DSS + +If you want to use the streamis0.2.0 front end normally, you also need to install the DSS streamisappconn plug-in. Please refer to: [streamisappconn plug-in installation document](development/StreamisAppConnInstallationDocument.md) + +## 6. Linkis Flink engine compilation and installation +If you want to run streamis0.2.0 normally, you also need to install the linkis Flink engine. Please refer to: [linkis Flink engine installation document](https://linkis.apache.org/zh-CN/docs/1.1.2/engine_usage/flink/) + +## 7. Streamis component upgrade document / script +If you want to upgrade from a lower version of streamis to streamis0.2.0, please refer to: [streamis upgrade document](development/StreamisUpgradeDocumentation.md) diff --git a/docs/en_US/0.2.0/development/StreamisAppConnInstallationDocument.md b/docs/en_US/0.2.0/development/StreamisAppConnInstallationDocument.md new file mode 100644 index 000000000..dcee32288 --- /dev/null +++ b/docs/en_US/0.2.0/development/StreamisAppConnInstallationDocument.md @@ -0,0 +1,73 @@ +Streamisappconn installation document this article mainly introduces the deployment, configuration and installation of streamisappconn in DSS (datasphere studio) 1.1.0 + +# 1. Preparation for deploying streamisappconn +Before deploying streamisappconn, please complete the installation of streamis0.2.0 and other related components, and ensure that the basic functions of the project are available. + +# 2. Download and compilation of streamisappconn plug-in +1) Download binary package + +We provide the material package of streamisappconn, which you can download directly. [Click to jump to the release interface](https://github.com/WeBankFinTech/Streamis/releases) + +2) Compile package + +If you want to develop and compile streamisappconn yourself, the specific compilation steps are as follows: 1 Code for clone streams; 2 Find the streamis appconn module and compile streamis appconn separately +```shell script +cd {STREAMIS_CODE_HOME}/streamis-appconn +mvn clean install +``` +Streamis will be found under this path Zip installation package +```shell script +{STREAMIS_CODE_HOME}\streamis-appconn\target\streamis.zip +``` + +# 3. Overall steps for deployment and configuration of streamisappconn plug-in +1. get the packed streamis Zip material package + +2. place it in the following directory and unzip it + +Note: after extracting streamis appconn for the first time, make sure that there is no index in the current folder_ v0000XX. Index file, which will be generated later +```shell script +cd {DSS_Install_HOME}/dss/dss-appconns +unzip streamis.zip +``` +The extracted directory structure is: +```shell script +conf +db +lib +``` +3. execute scripts for automatic installation + ```shell script +cd {DSS_INSTALL_HOME}/dss/bin +sh ./appconn-install.sh +# Script is an interactive installation scheme. You need to enter the string streamis and the IP and port of streamis service to complete the installation +# The streamis port here refers to the front-end port, which is configured in nginx. Instead of the back-end service port +``` + +## 4. After the installation of streamis appconn, you need to restart the DSS service to finally complete the plug-in update +###4.1) make the deployed appconn effective +Use the DSS start / stop script to make appconn effective. Enter the directory {DSS_INSTALL_HOME}/dss/sbin where the script is located, and execute the script using the following commands in sequence: +```shell script +sh ./dss-stop-all.sh +sh ./dss-start-all.sh +``` +###4.2) verify whether streamis appconn is effective +After installing and deploying streamis appconn, you can preliminarily verify whether the streamis appconn is successfully installed through the following steps. + +Create a new project in the DSS workspace +![DSS_workspace_Streamis_project](../../../images/zh_CN/dss_streamis_project.png) + +Check whether the project is created synchronously in the streamis database. Query the records indicating that appconn is successfully installed +```roomsql +SELECT * FROM linkis_stream_project WHERE name = ' input project_name '; +``` + +# 5. Installation principle of streamis appconn +The relevant configuration information of streamis will be inserted into the following table. The use configuration of streamis can be completed by configuring the following table. (Note: if you only need to quickly install appconn, you don't need to pay too much attention to the following fields. Most of the init.sql provided are configured by default. Focus on the above operations.) + +|table name |table function |remarks | +|-------------------|-----------------------------------------|------| +|dss_workspace_dictionary |Configuring a streaming production center |must| +|dss_appconn |Basic information of appconn, used to load appconn |must| +|dss_workspace_menu_appconn |Appconn menu, front-end connection to streamis |must| +|dss_appconn_instance |The instance information of appconn, including its own URL information |must| diff --git a/docs/en_US/userManual/StreamisUserManual.md b/docs/en_US/userManual/StreamisUserManual.md index e7e6982d5..4ebe42a8d 100644 --- a/docs/en_US/userManual/StreamisUserManual.md +++ b/docs/en_US/userManual/StreamisUserManual.md @@ -2,7 +2,7 @@ ## 1. Preface -         This article is a quick start document for Streamis 0.1.0, which covers the basic usage process of Stremis. More details on operation and usage will be provided in the user documentation. +         This article is a quick start document for Streamis 0.2.0, which covers the basic usage process of Stremis. More details on operation and usage will be provided in the user documentation. ## 2. Streamis entrance @@ -10,7 +10,7 @@ The entry path is **Home-DSS component application-Enter Streamis** -![Streamis entrance](../../images/entry_en.png) +![Streamis entrance](../../images/create_stream_product_center.png)
Picture 2.1 Streamis entrance]
## 3. Core indicators @@ -19,7 +19,7 @@ The entry path is **Home-DSS component application-Enter Streamis**          The core indicator shows the status summary of the Flink tasks uploaded to the project for execution. There are temporarily 7 states, showing the state name and the number of tasks in that state. The specific content is as shown in the figure below. -![Core indicators](../../images/homePage_en.png) +![Core indicators](../../images/home_page.png)
Picture 3.1 Core indicators
# 4. Job management @@ -88,48 +88,30 @@ The entry path is **Home-DSS component application-Enter Streamis**          The task list provides the function of **"query"**, where **job name** provides the function of fuzzy query. -## 4.2. Job details - -
- -         Click **"Version"** of a task in the task list to provide the function of viewing all versions of the task, as shown in the figure below: - -![Version details](../../images/versionDetail_en.png) -
Picture 4.2 Version details
- -
- -         Click "Open" of a certain version in the task list to view the **"Execution History"** and **"Task Details"** of that version, as shown in the figure below: - -![Version details](../../images/versionHistory_en.png) -
Picture 4.3 Version details
-
- -         Click **"Real-time Production Center"** in the figure above to return to the Streamis homepage. - ## 4.3. Management jobs
         Click **"job name"** of a task in the task list to provide the function of managing the task, or click **"three dots"** to the left of the name to call up the specific function configuration entry, as shown below: -![Configuration job](../../images/jobNav_en.png) +![Configuration job](../../images/job_list.png)
Picture4.3 Configuration job

   The configuration task provides four functions, namely: -- Job summary -- Job history -- Job config -- Job detail +- Parameter configuration +- alarm configuration +- operation history +- operation log +- snapshot[savepoint]
### 4.3.1. Job summary: -![Operating condition](../../images/statusDetail_en.png) +![Operating condition](../../images/stream_job_detail.png)
Picture 4.4 Job summary

@@ -140,7 +122,7 @@ The entry path is **Home-DSS component application-Enter Streamis** ### 4.3.2. Job history: -![Execution history](../../images/history_en.png) +![Execution history](../../images/stream_job_history.png)
Picture 4.5 Job history

@@ -152,7 +134,8 @@ The entry path is **Home-DSS component application-Enter Streamis** ### 4.3.3. Job config: -![Configuration](../../images/config_en.png) +![Configuration](../../images/stream_job_config_1.png) +![Configuration](../../images/stream_job_config_2.png)
Picture 4.6 Job config

@@ -162,7 +145,6 @@ The entry path is **Home-DSS component application-Enter Streamis** - Resource configuration - Production configuration - Flink parameters -- Alarm settings - Permission settings
@@ -177,7 +159,7 @@ The entry path is **Home-DSS component application-Enter Streamis** **Flink Jar Job details** -![Job details](../../images/jarDetail_en.png) +![Job details](../../images/stream_job_flinkjar_jobcontent.png)
Picture 4.7 Flink Jar Job details

@@ -189,7 +171,7 @@ The entry path is **Home-DSS component application-Enter Streamis** **Flink SQL job details** -![Job details](../../images/SQLDetail_en.png) +![Job details](../../images/stream_job_flinksql_jobcontent.png)
Picture 4.8 Flink SQL job details

@@ -210,10 +192,10 @@ The entry path is **Home-DSS component application-Enter Streamis**
-![Engineering Documents Home Page](../../images/fileHome_en.png) +![Engineering Documents Home Page](../../images/project_source_file_list.png)
Picture 5.1 Engineering Documents Home Page

-![Upload project file](../../images/fileUpload_en.png) +![Upload project file](../../images/project_source_file_import.png)
Picture 5.2 Upload project file
diff --git a/docs/images/create_script_file.png b/docs/images/create_script_file.png new file mode 100644 index 000000000..ce3030df9 Binary files /dev/null and b/docs/images/create_script_file.png differ diff --git a/docs/images/create_stream_product_center.png b/docs/images/create_stream_product_center.png new file mode 100644 index 000000000..701b4c683 Binary files /dev/null and b/docs/images/create_stream_product_center.png differ diff --git a/docs/images/enter_flinksql.png b/docs/images/enter_flinksql.png new file mode 100644 index 000000000..c360dbe28 Binary files /dev/null and b/docs/images/enter_flinksql.png differ diff --git a/docs/images/flinksql_job_use_demo.png b/docs/images/flinksql_job_use_demo.png new file mode 100644 index 000000000..7c8dd715d Binary files /dev/null and b/docs/images/flinksql_job_use_demo.png differ diff --git a/docs/images/flinksql_job_use_demo2.png b/docs/images/flinksql_job_use_demo2.png new file mode 100644 index 000000000..606fe116f Binary files /dev/null and b/docs/images/flinksql_job_use_demo2.png differ diff --git a/docs/images/flinksql_script_file.png b/docs/images/flinksql_script_file.png new file mode 100644 index 000000000..47229ab8c Binary files /dev/null and b/docs/images/flinksql_script_file.png differ diff --git a/docs/images/home_page.png b/docs/images/home_page.png new file mode 100644 index 000000000..80c419d53 Binary files /dev/null and b/docs/images/home_page.png differ diff --git a/docs/images/image-20211230173334826.png b/docs/images/image-20211230173334826.png deleted file mode 100644 index 5dc7f981c..000000000 Binary files a/docs/images/image-20211230173334826.png and /dev/null differ diff --git a/docs/images/image-20211230173621711.png b/docs/images/image-20211230173621711.png deleted file mode 100644 index 287b9f17d..000000000 Binary files a/docs/images/image-20211230173621711.png and /dev/null differ diff --git a/docs/images/image-20211231100801179.png b/docs/images/image-20211231100801179.png deleted file mode 100644 index 1fdead8dc..000000000 Binary files a/docs/images/image-20211231100801179.png and /dev/null differ diff --git a/docs/images/image-20211231101738880.png b/docs/images/image-20211231101738880.png deleted file mode 100644 index 50dda3641..000000000 Binary files a/docs/images/image-20211231101738880.png and /dev/null differ diff --git a/docs/images/jarDetail.png b/docs/images/jarDetail.png deleted file mode 100644 index 378a4a3b8..000000000 Binary files a/docs/images/jarDetail.png and /dev/null differ diff --git a/docs/images/job_list.png b/docs/images/job_list.png new file mode 100644 index 000000000..50fb4e4c9 Binary files /dev/null and b/docs/images/job_list.png differ diff --git a/docs/images/jobbulk_operate.png b/docs/images/jobbulk_operate.png new file mode 100644 index 000000000..38aabe699 Binary files /dev/null and b/docs/images/jobbulk_operate.png differ diff --git a/docs/images/project_source_file_import.png b/docs/images/project_source_file_import.png new file mode 100644 index 000000000..a7b2404b8 Binary files /dev/null and b/docs/images/project_source_file_import.png differ diff --git a/docs/images/project_source_file_list.png b/docs/images/project_source_file_list.png new file mode 100644 index 000000000..ebcf25982 Binary files /dev/null and b/docs/images/project_source_file_list.png differ diff --git a/docs/images/stream_job_config_1.png b/docs/images/stream_job_config_1.png new file mode 100644 index 000000000..5e391bf84 Binary files /dev/null and b/docs/images/stream_job_config_1.png differ diff --git a/docs/images/stream_job_config_2.png b/docs/images/stream_job_config_2.png new file mode 100644 index 000000000..a1ea87288 Binary files /dev/null and b/docs/images/stream_job_config_2.png differ diff --git a/docs/images/stream_job_detail.png b/docs/images/stream_job_detail.png new file mode 100644 index 000000000..99dcb0a97 Binary files /dev/null and b/docs/images/stream_job_detail.png differ diff --git a/docs/images/stream_job_flinkjar_jobcontent.png b/docs/images/stream_job_flinkjar_jobcontent.png new file mode 100644 index 000000000..9b0063eb3 Binary files /dev/null and b/docs/images/stream_job_flinkjar_jobcontent.png differ diff --git a/docs/images/stream_job_flinksql_jobcontent.png b/docs/images/stream_job_flinksql_jobcontent.png new file mode 100644 index 000000000..cdc9eceae Binary files /dev/null and b/docs/images/stream_job_flinksql_jobcontent.png differ diff --git a/docs/images/stream_job_history.png b/docs/images/stream_job_history.png new file mode 100644 index 000000000..c90a486a1 Binary files /dev/null and b/docs/images/stream_job_history.png differ diff --git a/docs/images/stream_product_center.png b/docs/images/stream_product_center.png new file mode 100644 index 000000000..a039e76c5 Binary files /dev/null and b/docs/images/stream_product_center.png differ diff --git a/docs/images/upload_jobtask_error.png b/docs/images/upload_jobtask_error.png new file mode 100644 index 000000000..ffa2033ae Binary files /dev/null and b/docs/images/upload_jobtask_error.png differ diff --git a/docs/images/upload_jobtask_error_solve.png b/docs/images/upload_jobtask_error_solve.png new file mode 100644 index 000000000..72971929e Binary files /dev/null and b/docs/images/upload_jobtask_error_solve.png differ diff --git "a/docs/images/\346\265\201\345\274\217\344\275\234\344\270\232\350\277\220\350\241\214\346\203\205\345\206\265.png" "b/docs/images/\346\265\201\345\274\217\344\275\234\344\270\232\350\277\220\350\241\214\346\203\205\345\206\265.png" deleted file mode 100644 index 57183a94d..000000000 Binary files "a/docs/images/\346\265\201\345\274\217\344\275\234\344\270\232\350\277\220\350\241\214\346\203\205\345\206\265.png" and /dev/null differ diff --git "a/docs/images/\346\265\201\345\274\217\345\272\224\347\224\250\351\205\215\347\275\256.png" "b/docs/images/\346\265\201\345\274\217\345\272\224\347\224\250\351\205\215\347\275\256.png" deleted file mode 100644 index 86c97d01f..000000000 Binary files "a/docs/images/\346\265\201\345\274\217\345\272\224\347\224\250\351\205\215\347\275\256.png" and /dev/null differ diff --git "a/docs/images/\347\224\237\344\272\247\344\270\255\345\277\203.png" "b/docs/images/\347\224\237\344\272\247\344\270\255\345\277\203.png" deleted file mode 100644 index 74b8d30da..000000000 Binary files "a/docs/images/\347\224\237\344\272\247\344\270\255\345\277\203.png" and /dev/null differ diff --git "a/docs/zh_CN/0.2.0/Streamis\345\256\211\350\243\205\346\226\207\346\241\243.md" "b/docs/zh_CN/0.2.0/Streamis\345\256\211\350\243\205\346\226\207\346\241\243.md" index f1e420fe9..8633b6c99 100644 --- "a/docs/zh_CN/0.2.0/Streamis\345\256\211\350\243\205\346\226\207\346\241\243.md" +++ "b/docs/zh_CN/0.2.0/Streamis\345\256\211\350\243\205\346\226\207\346\241\243.md" @@ -189,7 +189,7 @@ server { } ``` -5.加载nginx配置 +4.加载nginx配置 ```bash sudo nginx -s reload @@ -200,4 +200,7 @@ sudo nginx -s reload 如您想正常使用 Streamis0.2.0 前端,还需安装 DSS StreamisAppConn 插件,请参考: [StreamisAppConn 插件安装文档](development/StreamisAppConn安装文档.md) ## 6.Linkis Flink引擎编译安装 -如您想正常执行 Streamis0.2.0,还需安装 Linkis Flink 引擎,请参考: [Linkis Flink 引擎安装文档](development/Flink引擎编译文档.md) \ No newline at end of file +如您想正常执行 Streamis0.2.0,还需安装 Linkis Flink 引擎,请参考: [Linkis Flink 引擎安装文档](https://linkis.apache.org/zh-CN/docs/1.1.2/engine_usage/flink/) + +## 7.Streamis组件升级文档/脚本 +如您想从Streamis较低版本升级到 Streamis0.2.0 ,请参考:[Streamis升级文档](development/Streamis升级文档.md) \ No newline at end of file diff --git "a/docs/zh_CN/0.2.0/development/Flink\345\274\225\346\223\216\347\274\226\350\257\221\346\226\207\346\241\243.md" "b/docs/zh_CN/0.2.0/development/Flink\345\274\225\346\223\216\347\274\226\350\257\221\346\226\207\346\241\243.md" deleted file mode 100644 index 18bf85efa..000000000 --- "a/docs/zh_CN/0.2.0/development/Flink\345\274\225\346\223\216\347\274\226\350\257\221\346\226\207\346\241\243.md" +++ /dev/null @@ -1,54 +0,0 @@ -# Flink引擎编译文档 - -本文主要介绍在Linkis1.0.2中,Flink引擎的编译 - -### 1、Flink引擎的配置和部署 - -Linkis1.0.2支持的Flink版本是Flink1.12.2,理论上Linkis1.x 可以支持各个版本的Flink,但是Flink个版本之间的API差异太大,可能需要您按照API的变化修改Linkis中Flink引擎的代码并重新编译。 - -### 2、Flink engineConn的部署和加载 - -Linkis Flink引擎默认在Linkis1.0.2中是不会安装的,需要您手动进行编译并安装。 - -单独编译Flink引擎的方式 - -``` -${linkis_code_dir}/linkis-enginepconn-lugins/engineconn-plugins/flink/ -mvn clean install -``` - -编译完成后,您可以在${linkis_code_dir}/linkis-enginepconn-lugins/engineconn-plugins/flink/target目录下获取到Flink引擎的引擎包flink-engineconn.zip - -然后将其部署到 - -``` -${LINKIS_HOME}/lib/linkis-engineplugins -``` - -并将其解压,Flink引擎包的目录结构 - -``` -flink -├── dist -│   └── v1.12.2 -│   ├── conf -│   ├── conf.zip -│   ├── lib -│   └── lib.zip -└── plugin - └── 1.12.2 - └── linkis-engineconn-plugin-flink-1.0.2.jar -``` - -并重启linkis-engineplugin - -``` -cd ${LINKIS_HOME}/sbin -sh linkis-daemon restart cg-engineplugin -``` - -engineplugin更详细的介绍可以参看下面的文章。 -https://github.com/apache/incubator-linkis/wiki/EngineConnPlugin%E5%BC%95%E6%93%8E%E6%8F%92%E4%BB%B6%E5%AE%89%E8%A3%85%E6%96%87%E6%A1%A3#22-%E7%AE%A1%E7%90%86%E5%8F%B0configuration%E9%85%8D%E7%BD%AE%E4%BF%AE%E6%94%B9%E5%8F%AF%E9%80%89 - -需要注意的是,当您解压完Flink引擎包后,建议将flink-engineconn.zip删除,避免引擎启动后,将flink-engineconn.zip加载进物料库中。 - diff --git "a/docs/zh_CN/0.2.0/development/Streamis\347\274\226\350\257\221\346\226\207\346\241\243.md" "b/docs/zh_CN/0.2.0/development/Streamis\347\274\226\350\257\221\346\226\207\346\241\243.md" deleted file mode 100644 index 9a1c63251..000000000 --- "a/docs/zh_CN/0.2.0/development/Streamis\347\274\226\350\257\221\346\226\207\346\241\243.md" +++ /dev/null @@ -1,30 +0,0 @@ -# Streamis编译文档 - -本文主要介绍Streamis流式应用平台的编译,包括前端web和后端server。 - -### 1、Streamis代码编译 - -​ 从github上面获取源代码 - -​ 后台编译方式如下: - -``` -cd ${STREAMIS_CODE_HOME} -mvn -N install -mvn clean install -``` - -执行成功后会在工程的${STREAMIS_CODE_HOME}/assembly/target目录下生成安装包wedatasphere-streamis-0.1.0-dist.tar.gz - - 前端编译方式如下 - -``` -cd ${STREAMIS_CODE_HOME}/web -npm i -npm run build -``` - -编译成功后 - -在${STREAMIS_CODE_HOME}/web目录下生成dist.zip - diff --git "a/docs/zh_CN/0.2.0/\344\275\277\347\224\250\346\226\207\346\241\243/Streamis\347\224\250\346\210\267\346\211\213\345\206\214.md" "b/docs/zh_CN/0.2.0/\344\275\277\347\224\250\346\226\207\346\241\243/Streamis\347\224\250\346\210\267\346\211\213\345\206\214.md" index ee5818733..864178179 100644 --- "a/docs/zh_CN/0.2.0/\344\275\277\347\224\250\346\226\207\346\241\243/Streamis\347\224\250\346\210\267\346\211\213\345\206\214.md" +++ "b/docs/zh_CN/0.2.0/\344\275\277\347\224\250\346\226\207\346\241\243/Streamis\347\224\250\346\210\267\346\211\213\345\206\214.md" @@ -2,7 +2,7 @@ ## 1. 前言 -         本文是Streamis0.1.0的快速入门文档,涵盖了Stremis的基本使用流程,更多的操作使用细节,将会在用户使用文档中提供。 +         本文是Streamis0.2.0的快速入门文档,涵盖了Stremis的基本使用流程,更多的操作使用细节,将会在用户使用文档中提供。 ## 2. Streamis整合至DSS @@ -11,43 +11,26 @@ ##### 2.1 **如何接入?** -​ 在**DSS数据库**中执行以下SQL: - -​ **特别注意:** - -​ 需要将以下SQL中URL字段的值:http://127.0.0.1:9188/#/realtimeJobCenter?projectName=${projectName}&workspaceName=${workspaceName} - -中的IP(127.0.0.1)和端口(9188)替换成您部署的streamis的IP与端口,其他内容无需改动 - -``` -INSERT INTO `dss_dictionary` ( `workspace_id`, `parent_key`, `dic_name`, `dic_name_en`, `dic_key`, `dic_value`, `dic_value_en`, `title`, `title_en`, `url`, `url_type`,`icon`, `order_num`, `remark`, `create_user`, `create_time`, `update_user`, `update_time`) -VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center','pdp_streamis_product_center','streamis_prod',NULL,NULL,NULL, -'http://127.0.0.1:9188/#/realtimeJobCenter?projectName=${projectName}&workspaceName=${workspaceName}','0','kaifa-icon','1','工程开发流程-流式生产中心','SYSTEM','2020-12-28 17:32:35',NULL,'2021-12-30 17:49:02'); -``` +按照 [StreamisAppConn安装文档](../development/StreamisAppConn安装文档.md) 安装部署StreamisAppConn成功后,Streamis系统会自动嵌入DSS系统中。 ##### 2.2 如何验证 DSS 已经成功集成了 Streamis? -​ 请进入 DSS 的工程首页,创建一个工程 - -![image-20211230173334826](../../../images/image-20211230173334826.png) - -进入到工程里面,点击右上角"开发中心"按钮切换到”流式生产中心“ +请进入 DSS 的工程首页,创建一个工程 -![image-20211230173621711](../../../images/image-20211230173621711.png) +![image-20211230173334826](../../../images/create_stream_product_center.png) -如果出现streamis的首页,则表示则表示 DSS 已经成功集成了 Streamis。如下图: - -![image-20211230173839138](../../../images/生产中心.png) +进入到工程里面,点击左上角按钮切换到”流式生产中心“,如果出现streamis的首页,则表示 DSS 已经成功集成了 Streamis。如下图: +![image-20211230173839138](../../../images/stream_product_center.png) ## 3. 核心指标 -         进入到首页,上半部显示的是核心指标。 +进入到streamis首页,上半部显示的是核心指标。 -         核心指标显示上传到该项目执行的Flink任务的状态汇总,状态暂时有7种,显示状态名称和处于该状态的任务数量,具体内容如下图。 +核心指标显示当前用户可查看到的上传到该项目执行的Flink任务的状态汇总,状态暂时有9种,显示状态名称和处于该状态的任务数量,具体内容如下图。 -![核心指标](../../../images/homePage.png) +![核心指标](../../../images/home_page.png)
图 3.1 首页核心指标
@@ -57,17 +40,17 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center',' ## 4.1. Script开发SQL -       创建一个脚本文件,脚本类型选择Flink,如下图所示: +       顶部Scriptis菜单创建一个脚本文件,脚本类型选择Flink,如下图所示: -![image-20211230174445688](../../../images/image-20211230174445688.png) +![进入FlinkSQL](../../../images/enter_flinksql.png) -编写FlinkSQL,source,sink,transform等。 +![create_script_file.png](../../../images/create_script_file.png) -![image-20211230174723105](../../../images/image-20211230174723105.png) +编写FlinkSQL,source,sink,transform等。 -点击运行后,即可调试该脚本,点击任务管理器可看到当前运行的引擎 +![flinksql_script_file](../../../images/flinksql_script_file.png) -![image-20211230175424588](../../../images/image-20211230175424588.png) +点击运行后,即可调试该脚本 ## 4.2. 发布至Streamis @@ -128,19 +111,32 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center',' ### 4.2.2 示例 -​ dzshengjiavars为flinksql文件,meta.json是该任务的元数据信息。 +​ streamisjobtest为flinksql文件,meta.json是该任务的元数据信息。 -![image-20211231092520768](../../../images/image-20211231092520768.png) +![flinksql_job_use_demo](../../../images/flinksql_job_use_demo.png)
-![image-20211231092959561](../../../images/image-20211231092959561.png) +![flinksql_job_use_demo2](../../../images/flinksql_job_use_demo2.png) 将SQL文件和meta.json文件打包成一个zip文件,注意:只能打包成zip文件,其他格式如rar、7z等格式无法识别。 +如果上传zip文件出现下面错误,请调整下nginx的配置`vi /etc/nginx/conf.d/streamis.conf`,添加属性`client_max_body_size`,如下图所示。 +![upload_jobtask_error](../../../images/upload_jobtask_error.png) +![upload_jobtask_error_solve](../../../images/upload_jobtask_error_solve.png) +----- + 在streamis中将该zip包导入,导入任务后,任务的运行状态变成"未启动",版本会+1(导入新的job任务版本从1开始),最新发布时间会更新至最新时间。 -![image-20211231094103002](../../../images/image-20211231094103002.png) +点击相应的作业名称、配置或左边3个竖点中(参数配置/告警配置/运行历史/运行日志)可进入job任务详情,点击 启动 可执行作业。 + +点击左边3个竖点中 快照【savepoint】 可保存快照。 + +![job_list](../../../images/job_list.png) + +点击批量操作,可选中多个作业任务重启,快照重启会先生成快照再重新启动,直接重启不会生成快照 + +![jobbulk_operate](../../../images/jobbulk_operate.png) #### @@ -151,7 +147,7 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center',' ## 5.1 运行情况 -​ 实时数据展示待二期开发。![image-20211231100653747](../../../images/流式作业运行情况.png) +![stream_job_detail](../../../images/stream_job_detail.png) ## 5.2 执行历史 @@ -161,17 +157,16 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center',' 历史日志中可以查看当前任务启动的flink引擎的日志,可以根据关键字等查看关键日志,点击查看最新日志,可以查看当前引擎的最新日志。 - - -![image-20211231100801179](../../../images/image-20211231100801179.png) - -![](../../../images/image-20211231101048962.png) +![stream_job_history](../../../images/stream_job_history.png) ## 5.3 配置 给Streamis任务配置一些flink资源参数以及checkpoint的参数 -![image-20211231101503678](../../../images/流式应用配置.png) +![image-20211231101503678](../../../images/stream_job_config_1.png) +![image-20211231101503678](../../../images/stream_job_config_2.png) + + ## 5.4任务详情 @@ -181,25 +176,17 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center','
-**Flink Jar任务详情** +- **Flink Jar任务详情** -![任务详情](../../../images/jarDetail.png) - -
Flink Jar任务详情
- -
+![任务详情](../../../images/stream_job_flinkjar_jobcontent.png)   Flink Jar任务详情展示了任务Jar包的内容和参数, 同时提供下载该Jar包的功能。
-**Flink SQL任务详情** - -![image-20211231101738880](../../../images/image-20211231101738880.png) +- **Flink SQL任务详情** -
Flink SQL任务详情
- -
+![任务详情](../../../images/stream_job_flinksql_jobcontent.png)   Flink SQL任务详情展示了该任务的SQL语句。 @@ -211,3 +198,12 @@ VALUES ('0','p_develop_process','流式生产中心','Streamis Product Center',' ![image-20211231102020703](../../../images/image-20211231102020703.png) +## 6 工程资源文件 +Streamis首页-核心指标右上角-工程资源文件。 +工程资源文件提供了上传和管理项目所需资源文件的功能,如下图所示: + +![project_source_file_list](../../../images/project_source_file_list.png) + +上传项目文件 + +![project_source_file_import](../../../images/project_source_file_import.png)