Streams in Node.js

Streams are a fundamental concept in Node.js for handling reading or writing data sequentially. They are particularly useful for working with large amounts of data or data that arrives over time (like network requests) because they allow you to process data in chunks rather than loading everything into memory at once.

Key Characteristics of Streams:

• Abstract Interface: Streams provide a unified way to work with different kinds of I/O sources and destinations (files, HTTP requests/responses, TCP sockets, child process stdio).
• Efficiency: By processing data in chunks, streams are memory-efficient.
• Composability: Streams can be pipe()d together to create powerful data processing pipelines.

Types of Streams:

1. Readable Streams: Represent a source from which data can be read (e.g., fs.createReadStream(), an HTTP server's request object, an HTTP client's response object).

   • You read from them by listening to events like data and end.

2. Writable Streams: Represent a destination to which data can be written (e.g., fs.createWriteStream(), an HTTP server's response object, an HTTP client's request object, process.stdout).

   • You write to them using methods like write() and end().

3. Duplex Streams: Are both Readable and Writable (e.g., a TCP socket, zlib streams for compression/decompression).

4. Transform Streams: Are Duplex streams where the output is a transformation of the input (e.g., a stream that converts text to uppercase, a compression stream).

Working with Writable Streams:

Objects that are Writable Streams typically have:

• write(chunk, [encoding], [callback]): Writes a chunk of data (can be a string or a Buffer). The optional callback is invoked when this chunk of data has been flushed.

• end([chunk], [encoding], [callback]): Signals that no more data will be written to the stream. An optional final chunk can be written before closing. The callback is invoked when the stream has finished.

Example: fs.createWriteStream() creates a Writable Stream to a file. You can use its write() method to write data to the file in pieces.

Working with Readable Streams:

Reading from a Readable Stream is typically event-driven:

• 'data' event: Emitted whenever a chunk of data is available to be read from the stream. The event handler receives the chunk (usually a Buffer or a string if an encoding was set).

• 'end' event: Emitted when there is no more data to be read from the stream.

• 'error' event: Emitted if an error occurs while reading.

• on(eventName, callback): Node.js event-emitting objects (including streams) have an on() method (similar to addEventListener in browsers) to register event handlers.

Example: The request object in an HTTP server's request handler and the response object in an HTTP client's callback are Readable Streams. fs.createReadStream() creates a Readable Stream from a file.

Example: Server Reading Request Body and Responding in Uppercase (Streaming)

This server reads the body of an incoming request and streams back the same content, but converted to uppercase.

const http = require('http');

http.createServer(function (request, response) {
    // Check if the request method is POST, as GET requests typically don't have a body
    if (request.method === 'POST') {
        response.writeHead(200, { 'Content-Type': 'text/plain' });

        // The 'request' object is a Readable Stream representing the incoming request body.
        // Listen for 'data' events to receive chunks of the request body.
        request.on('data', function (chunk) {
            // 'chunk' is a Buffer object. Convert it to a string and then to uppercase.
            // The 'response' object is a Writable Stream. Write the transformed chunk.
            response.write(chunk.toString().toUpperCase());
        });

        // Listen for the 'end' event, which signifies the entire request body has been received.
        request.on('end', function () {
            // End the response stream once all data has been processed and sent.
            response.end();
            console.log('Finished processing POST request.');
        });

        request.on('error', (err) => {
            console.error('Error reading request body:', err);
            response.statusCode = 400; // Bad Request
            response.end('Error processing request body.');
        });

    } else if (request.method === 'GET') {
        response.writeHead(200, { 'Content-Type': 'text/html' });
        response.end('<h1>Send a POST request with a body to see streaming in action!</h1><p>Example using curl: <code>curl -X POST -d "hello world" http://localhost:8000</code></p>');
    } else {
        response.writeHead(405); // Method Not Allowed
        response.end(`Method ${request.method} not allowed.`);
    }
}).listen(8000, () => {
    console.log('Server listening on port 8000 for POST requests...');
});

Example: Client Sending Data and Receiving Uppercase Response (Streaming)

This client sends a POST request to the server above and prints the streamed uppercase response.

const http = require('http');

const options = {
    hostname: 'localhost',
    port: 8000,
    method: 'POST',
    headers: {
        'Content-Type': 'text/plain' // Informing server about the type of data we're sending
    }
};

const clientRequest = http.request(options, function (serverResponse) {
    // 'serverResponse' is a Readable Stream representing the server's response.
    console.log('Server responded with status:', serverResponse.statusCode);

    // Listen for 'data' events to receive chunks of the server's response.
    serverResponse.on('data', function (chunk) {
        // process.stdout is a Writable Stream representing the standard output of the process.
        // We use it instead of console.log to avoid extra newlines per chunk.
        process.stdout.write(chunk.toString());
    });

    serverResponse.on('end', function () {
        process.stdout.write('\n--- Server response fully received ---\n');
    });

    serverResponse.on('error', (err) => {
        console.error('Error in server response:', err);
    });
});

clientRequest.on('error', (err) => {
    console.error('Error making the request:', err.message);
});

// 'clientRequest' is a Writable Stream. Write the request body.
clientRequest.write('Hello server, this is a streamed message.');
clientRequest.write(' And this is another part of the message.');

// End the request, signaling that we've sent all data for the request body.
clientRequest.end(); // You can also do request.end("Hello server") for a single chunk.
console.log('POST request sent to server...');

In this client example:

• process.stdout.write(chunk.toString()) is used instead of console.log(chunk.toString()).
  • console.log() automatically adds a newline character (\n) after each output.
  • process.stdout.write() writes the data directly without adding any extra characters, which is more suitable when assembling a streamed response chunk by chunk.

Streams are a powerful and efficient way to handle I/O in Node.js, especially for scenarios involving large data or real-time data transfer.