Jay Gould

Creating a dynamic loading bar with Socket.io and Node.js

October 24, 2020

My last post listed attempts I made at speeding up long running server side processes in Node, but as well as ensuring a process is fast, an app also needs to give a good user experience. A huge part of user experience when waiting for a computer to process something is keeping the user informed about how long the process will take. This post expands on the web app from my last post by adding in a progress bar created with Socket.io.

Progress loading bar

The problem

Once the user has submitted the form on the front end, there’s no more communication coming back to the front end by default to give an update on the progress of the processing on the server. My app can take large file size uploads from users, and as mentioned in my last post, times of a minute or so or processing in Node.js are expected. The goal is to try and keep the user updated with a percentage based progress component.

This happens by the client being updated on the progress from the server - by either the client sending requests to get a progress update, or by the server sending updates back to the client. The progress will need to be calculated at some point while the data is being worked on, so finding how and when to do this is crucial.

In order to get this communication between client and server, one way would be to use long polling whereby a normal HTTP request to the server is kept open until the server sends something back to the client, however this can take up a fair amount of server resources per connection, which is something I want to stay away from on my already limited EC2 instance. A better alternative I think is to use WebSockets, which is a different protocol than HTTP, and purpose made for a constant connection on which both client and server can initiate requests on independently.

Setting up WebSockets with Socket.io

The first step is to get Socket.io installed on the front end:

npm i socket.io-client

Once installed, it can be used by integrating in to a component:

// Client

import React, { useState, useEffect } from "react"
import io from "socket.io-client"

function Dashboard() {
  const [uploadProgress, setUploadProgress] = useState(null)

  useEffect(() => {
    const socket = io("http://localhost:3000/api")
    socket.on("uploadProgress", (data) => {
      setUploadProgress(data)
    })
  }, [])

  return <div>{uploadProgress ? uploadProgress : null}</div>
}

The above is using a React functional component with Hooks to manage and update the progress state on the front end. The io() function is taking the URL of the API to create a connection, and then using socket.on() to listen to updates from that connection. Once an update is received from the server, the setUploadProgress() function updates the state via the Hook, and displays the progress in the page.

Adding socket.io to the server

The server now needs to be configured to accept the connection initiated by the front end. Start by installing Socket.io on the server:

npm i socket.io

Once installed, the server can be set up to accept incoming connections:

// Server

import { app } from "./app"

const sockets = {}

const server = app.listen(app.get("port"), () => {})

const io = require("socket.io")(server)

io.on("connection", (socket) => {
  console.log(`Client connected: ${socket.id}`)
})

// The io instance is set in Express so it can be grabbed in a route
app.set("io", io)

The server is set up so when a connection is initiated, the socket ID is logged. This is the basic setup which will be enough to show there’s a connection between client and server. Once a connection is made, both client and server can send messages to one another, but this is not as easy as it might seem because the server needs to know which client socket ID to send messages back to.

Sending messages back to a specific socket ID

The socket ID’s on the server can be stored in memory, but the server also needs to know which ID to communicate with for certain actions. A nice solution I found to do this is to generate a random ID on the client each time the client submits a form:

// Client

const thisSessionId = Math.random().toString(36).substr(2, 9);

function Dashboard() {
  const [uploadProgress, setUploadProgress] = useState(null);

  useEffect(() => {
  const socket = io('http://localhost:3000/api);
    // Initiates the connection when the user visits the page, sending up the clients
    // unique ID
    socket.emit('connectInit', thisSessionId);
    socket.on('uploadProgress', (data) => {
      setUploadProgress(data);
    });
  }, []);
}

This thisSessionId variable is sent in the form submission along with the file upload, and stored in memory:

// Server

const sockets = {}

io.on("connection", (socket) => {
  socket.on("connectInit", (sessionId) => {
    // The socket ID is stored along with the unique ID generated by the client
    sockets[sessionId] = socket.id
    // The sockets object is stored in Express so it can be grabbed in a route
    app.set("sockets", sockets)
  })
})

// The io instance is set in Express so it can be grabbed in a route
app.set("io", io)

As well as storing the socket ID’s in memory, other storage solutions can be used. One perfect solution for this is Redis which is out of scope of this post, but worth implementing for reasons I mention at the bottom of this post.

The next step is to get the server to send a message back to the client to update on the progress, using the socket ID that’s being stored.

Sending a progress update to the client using a socket connection

The progress update needs to be integrated directly in to the process and emitted from the server after a certain amount of data has processed. As explained in the last post, the data upload is processed in chunks on the server to keep the process time down, so a percentage calculation can be made and sent back after each chunk has been processed:

// Server

const uploadedAddresses = uploadedDataArray
const originLocationData = objectContainingLatLngOfOriginLocation
const searchRadius = 10

// Get the socket connection from Express app
const io = req.app.get('io');
const sockets = req.app.get('sockets');
const thisSocketId = sockets[socketSessionId];
const socketInstance = io.to(thisSocketId);
socketInstance.emit('uploadProgress', 'File uploaded, processing data...');

let chunksProcessed = 0;
let chunkSize = 100

// Split the uploaded array in to chunks so each chunk can be queried in one go
const uploadChunks = chunkArray(uploadedAddresses, chunkSize)

// Map over each chunk sequentially (1 concurrency), and get postcode data
const dataWithDbPostcode = await Promise.map(
  uploadChunks,
  async (uploadChunk: any) => {
    const dbChunk = await db.postcodes.findAll({
      where: {
        postcodeNoSpace: {
          [db.Sequelize.Op.in]: uploadChunk
        }
      }
    })

    // Calculate the progress based on the size of each chunk, the count of chunks being processed,
    // and the total rows in the file upload
    const progress = Math.round(((chunkSize * chunksProcessed) / totalRows) * 100 * 10) / 10;
    chunksProcessed++;

    // Send the progress calculation back to the client using the socket ID from earlier
    socketInstance.emit('uploadProgress', `${progress}%`);

    return dbChunk
  },
  { concurrency: 1 }
)

// Flatten the chunks of uploaded and queried postcodes and get the distance
const processedData = dataWithDbPostcode.flat().map(address => {
  const distance = distance(
    postcodeData.latitude,
	  postcodeData.longitude,
	  originLocationData.latitude,
	  originLocationData.longitude
  )

  return {
    coordinates: {
      latitude: address.dbRow.latitude,
      longitude: address.dbRow.longitude,
    },
    distance,
    isWithinRange: distance < this.radius ? "Yes" : "No",
  }
})

// Chunk array function
chunkArray(arr, chunk_size) {
  var results = [];

  while (arr.length) {
    results.push(arr.splice(0, chunk_size));
  }

  return results;
}

There’s a lot going on in that snippet, and it’s actually spread out over a couple of classes and different files in my actual app, but it’s together here to keep things simple. But essentially it’s following on from my last post but emitting a socket message after each chunk is processed.

Updating the client with the progress response

As the client and server have the socket connection set up, the client is sent the progress updates when a chunk is processed, so the front end UI can be updated to do anything when an update is received. In my example, this is done by updating React with setUploadProgress:

// Client

import React, { useState, useEffect } from "react"
import io from "socket.io-client"

function Dashboard() {
  const [uploadProgress, setUploadProgress] = useState(null)

  useEffect(() => {
    const socket = io("http://localhost:3000/api")
    socket.on("uploadProgress", (data) => {
      setUploadProgress(data) // Data from progress added to state
    })
  }, [])

  return (
    <div>
      {uploadProgress ? (
        <div className={css.progressBar}>
          <div
            className={css.progressBar__completed}
            style={{ width: `${uploadProgress.percentage}%` }}
          >
            <div className={css.progressBar__display}>
              {uploadProgress ? uploadProgress.display : null}
            </div>
          </div>
          <div className={css.progressBar__display}>
            {uploadProgress.percentage === 0 ? uploadProgress.display : null}
          </div>
        </div>
      ) : null}
    </div>
  )
}

As the progress percentage increases, the width of an inner div increases it’s percentage by the same amount to show the loading bar. Here’s the CSS to go with the component:

.progressBar {
  width: 100%;
  border: 1px solid #e2e2e2;
  height: 40px;
  position: relative;
  &__display {
    position: absolute;
    height: 100%;
    width: 100%;
    top: 0px;
    left: 0px;
    text-align: center;
    line-height: 40px;
    z-index: 30;
  }
  &__completed {
    background: linear-gradient(187deg, rgb(156 133 18 / 74%) 0%, #caaf4e 100%);
    color: #fff;
    width: 0%;
    position: absolute;
    top: 0px;
    left: 0px;
    height: 100%;
    z-index: 10;
    border-radius: 3px;
  }
}

Improvements

Although this works great, one issue is that when the server restarts (perhaps due to an error) the socket ID’s are all lost because the memory is wiped. One way to combat this is to store the socket ID’s in a Redis cache. The cache is persistent on server restarts, and while it’s not possible to continue processing a file upload where it left off, it’s great to inform the user from the server that the connection has been lost and to instruct the user to re-upload their file.

Senior Engineer at Haven

© Jay Gould 2023, Built with love and tequila.