- Use storage drop methods to properly clean up all data - Add file deletion from input directory - Add pipeline status checking and locking mechanism - Improve error handling with detailed logging and pipeline message tracking