Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to make chunk upload in parallel ? #187

Open
ghost opened this issue Oct 23, 2018 · 1 comment
Open

how to make chunk upload in parallel ? #187

ghost opened this issue Oct 23, 2018 · 1 comment

Comments

@ghost
Copy link

ghost commented Oct 23, 2018

Someone can help me, to know how I can do with dropzone to upload chunk is parallel. I have this component in react.


import React from 'react';
import DropzoneComponent from 'react-dropzone-component';
import request, { defaultHeaders } from 'utils/request';
import Modal from 'components/Atoms/Modal';
import Button from 'components/Atoms/Button';

import ModalFigureError from 'images/modal/modal-figure-error.svg';
import Styles from './documentPage.css';
export default class Example extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      showPlaceholder: true,
      filename: '',
      sliceSize: 262144,
      start: 0,
      totalChunks: 0,
      numRequest: 1,
      modalIsOpen: false,
      errorCopie: '',
      modalErrorReportIsOpen: false,
      errorReportLimit: {},
      errorObject: {}
    };

    this.djsConfig = {
      addRemoveLinks: true,
      autoProcessQueue: false,
      headers: defaultHeaders(true),
      maxFilesize: 2048,
      paramName: 'csv',
      params: {
        industry: 'ecommerce'
      },
      chunking: true,
      chunkSize: 262144,
      timeout: 60000,
      maxFiles: 1,
      parallelUploads: 5,
      acceptedFiles: '.csv',
      retryChunks: true,
      retryChunksLimit: 5
    };

    this.componentConfig = {
      iconFiletypes: ['.csv'],
      showFiletypeIcon: true,
      postUrl: `${process.env.apiUrl}/csv/upload`
    };

    this.dropzone = null;
  }

  handleAddedFile = (file) => {
    this.setState({
      errorCopie: ''
    });
    console.log(this.djsConfig);
    console.log(this.dropzone, 'this.dropzone');
    if (file.size <= 1048576) {
      this.djsConfig.chunking = false;
      this.dropzone.options.chunking = false;
    }
    request(`${process.env.apiUrl}/csv/filename`, {
      method: 'GET'
    }).then((res) => {
      this.setState({
        filename: res.filename,
        totalChunks: file.upload.totalChunkCount
      });
      this.dropzone.processQueue();
      console.time();
    });

    this.handleShowPlaceholder();
  };

  handleRemovedFile = () => {
    this.setState({ filename: '', start: 0, totalChunks: 0, numRequest: 1 });
    // this.props.handleHiddenButton();
  };

  handleShowPlaceholder = () => {
    this.setState({
      showPlaceholder: !this.state.showPlaceholder
    });
  };

  handleSendingFile = (file, xhr, formData) => {
    this.dropzone.processQueue();
    const { filename, sliceSize, start, totalChunks, numRequest } = this.state;
    let end = start + sliceSize;

    if (file.size - end < 0) {
      end = file.size;
    }

    formData.append('filename', filename);
    formData.append('start', start);
    formData.append('size', file.size);
    formData.append('total_chunks', totalChunks);
    formData.append('id_chunk', numRequest);
    console.log(numRequest, 'numRequest');

    if (end < file.size) {
      this.setState({
        start: this.state.start + sliceSize,
        numRequest: numRequest + 1
      });
    }
  };

  handleSuccessFile = (file) => {
    console.timeEnd();
    const { filename, totalChunks, numRequest } = this.state;
    if (numRequest === 2 || numRequest === totalChunks) {
      request(`${process.env.apiUrl}/csv/check_csv`, {
        method: 'POST',
        body: {
          industry: 'ecommerce',
          filename
        }
      })
        .then(() => {
          this.handleShowPlaceholder();
          localStorage.setItem('id_csv', this.state.filename);
        })
        .catch((error) => {
          this.dropzone.removeFile(file);
          this.handleShowPlaceholder();
          this.setState({
            errorCopie: error.message
          });
        });
    }
  };

  handleUploadprogress = (file, progress, bytesSent) => {
    let progressChunk = progress;
    progressChunk = (bytesSent / file.size) * 100; // esint-disable-line
    document.querySelector(
      '[data-dz-uploadprogress]'
    ).style.width = `${progressChunk}%`;
  };

  handleError = (error) => {
    let errorObject = {
      error:
        'An error occurred, please try again. If the problem persists please contact our support team.'
    };
    if (error.xhr.response && error.xhr.response !== '') {
      errorObject = JSON.parse(error.xhr.response);
    }
    this.setState({
      errorCopie: errorObject
    });

    if (errorObject.code === 'limit-size') {
      this.openModalmodalError();
    }
    if (errorObject.code === 'report_limit') {
      this.openModalReportError();
    }
  };

  goNext = () => {
    if (!this.state.showPlaceholder) {
      localStorage.setItem('id_csv', this.state.filename);
    }
  };

  openModalReportError = () => {
    this.setState({ modalErrorReportIsOpen: true });
  };

  closeModalReportError = () => {
    this.setState({ modalErrorReportIsOpen: false });
  };

  openModalmodalError = () => {
    this.setState({ modalErrorIsOpen: true });
  };

  closeModalmodalError = () => {
    this.setState({ modalErrorIsOpen: false });
  };

  renderModalError = () => (
    <Modal
      isOpen={this.state.modalErrorIsOpen}
      onRequestClose={this.closeModalmodalError}
    >
      <div className={Styles.Document__Modal}>
        <figure className={Styles.Document__Modal__Logo}>
          <img src={ModalFigureError} alt="Figure DataGran Error" />
        </figure>
        <div className={Styles.Document__Modal__Content}>
          <h2 className={Styles.Document__Modal__Title}>
            Your file is too big!
          </h2>
          <p className={Styles.Document__Modal__Text}>
            Your current BDA plan allows you to upload files up to{' '}
            {this.state.errorCopie.max_size}
            Mb so we are unable to complete this request. Please reduce the size
            and upload it again
          </p>
          <div className={Styles.Document__Modal__Action}>
            <Button onClick={this.closeModalmodalError}>Got it!</Button>
          </div>
        </div>
      </div>
    </Modal>
  );

  renderModalReportLimit = () => {
    const { errorCopie } = this.state;
    return (
      <Modal
        isOpen={this.state.modalErrorReportIsOpen}
        onRequestClose={this.closeModalReportError}
      >
        <div className={Styles.Document__Modal}>
          <figure className={Styles.Document__Modal__Logo}>
            <img src={ModalFigureError} alt="Figure DataGran Error" />
          </figure>
          <div className={Styles.Document__Modal__Content}>
            <h2 className={Styles.Document__Modal__Title}>
              You have reached your monthly limit!
            </h2>
            <p className={Styles.Document__Modal__Text}>
              Your current BDA plan allows you to create up to{' '}
              {errorCopie.max_reports} reports monthly and you have already
              reached that limit :(
            </p>
            <p className={Styles.Document__Modal__Text}>
              Remember that you can keep exploring and exporting audiences on
              all the reports you have already created.
            </p>
            <div className={Styles.Document__Modal__Action}>
              <Button onClick={this.closeModalReportError}>Got it!</Button>
            </div>
          </div>
        </div>
      </Modal>
    );
  };

  render() {
    const config = this.componentConfig;
    const djsConfig = this.djsConfig;

    // For a list of all possible events (there are many), see README.md!
    const eventHandlers = {
      init: (dz) => (this.dropzone = dz),
      addedfile: this.handleAddedFile,
      removedfile: this.handleRemovedFile,
      sending: this.handleSendingFile,
      success: this.handleSuccessFile,
      uploadprogress: this.handleUploadprogress,
      error: this.handleError
    };

    return (
      <DropzoneComponent
        config={config}
        eventHandlers={eventHandlers}
        djsConfig={djsConfig}
      />
    );
  }
}
@0x111
Copy link

0x111 commented Dec 21, 2018

Based on my observation, there is an option parallelChunkUploads for the dropzonejs component. Would that be helpful for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant