Python Programming
Reading a Growing Log File from the Last Read Position
In real-world applications, log files continuously grow as new entries are added.
Examples:
- Web server logs
- Application logs
- Security logs
- System logs
- Audit logs
A common requirement is:
Read only the newly added log entries instead of re-reading the entire file every time.
Python provides two important file methods for this:
file.tell() # Get current file position
file.seek() # Move to a specific position
Understanding File Positions
Consider a log file:
Server Started
User Login
Database Connected
Internally, Python keeps track of the current reading position.
0 -----------------------------> End of File
^
Current Position
When reading progresses:
Server Started
^^^^^^^^^^^^^^
Position Moves Forward
Python can:
- Remember the current position
- Save it
- Return to it later
This is the foundation of log monitoring systems.
Method 1: Read New Logs While Program Keeps Running
If your Python program never stops, you can continuously monitor the log file.
This works similarly to:
tail -f app.log
Example
with open("app.log", "r") as log_file:
# Move to end of file
log_file.seek(0, 2)
while True:
line = log_file.readline()
if not line:
continue
print("New Log:", line.strip())
How It Works
Step 1
Open the file:
with open("app.log", "r")
Step 2
Move to the end:
log_file.seek(0, 2)
Parameters:
seek(offset, reference)
Meaning:
| Value | Meaning |
|---|---|
| 0 | Beginning |
| 1 | Current Position |
| 2 | End of File |
So:
log_file.seek(0, 2)
means:
Move 0 bytes from the end
which places the cursor at the end of the file.
Step 3
Read one line:
line = log_file.readline()
Step 4
If no new line exists:
if not line:
continue
wait and check again.
Example
Initial log:
[INFO] Server Started
Program starts.
Nothing happens.
Later:
[INFO] User Login
gets added.
Output:
New Log: [INFO] User Login
Limitation of Method 1
This works only while the program remains running.
If the program stops:
Program Stopped
↓
Position Lost
When restarted:
Reading Starts Again
which may cause duplicate processing.
Method 2: Save Last Read Position
This is the most common solution in production systems.
Idea:
Read Log
↓
Save Current Position
↓
Program Stops
↓
Program Starts Again
↓
Continue From Saved Position
Using an Offset File
Create:
app.log
offset.txt
Step 1: Read Saved Position
import os
offset = 0
if os.path.exists("offset.txt"):
with open("offset.txt", "r") as f:
offset = int(f.read())
Explanation
Suppose:
offset.txt
contains:
150
Then:
offset = 150
means:
Continue reading from byte 150
Step 2: Move to That Position
with open("app.log", "r") as log:
log.seek(offset)
Example:
log.seek(150)
Cursor jumps directly to byte 150.
Step 3: Read Only New Entries
for line in log:
print(line.strip())
Only unread entries will be processed.
Step 4: Save New Position
After reading:
new_offset = log.tell()
Save it:
with open("offset.txt", "w") as f:
f.write(str(new_offset))
Understanding tell()
Example:
with open("app.log", "r") as log:
print(log.tell())
Output:
0
After reading:
log.read(50)
print(log.tell())
Output:
50
Meaning:
Current Position = 50 Bytes
Complete Example
import os
offset = 0
if os.path.exists("offset.txt"):
with open("offset.txt", "r") as f:
offset = int(f.read())
with open("app.log", "r") as log:
log.seek(offset)
for line in log:
print(line.strip())
new_offset = log.tell()
with open("offset.txt", "w") as f:
f.write(str(new_offset))
Example Walkthrough
Initial log:
Log1
Log2
Log3
First Run
Output:
Log1
Log2
Log3
Position:
15
Saved to:
offset.txt
Contents:
15
Later
Log file becomes:
Log1
Log2
Log3
Log4
Log5
Program starts again.
Reads:
offset = 15
Moves:
log.seek(15)
Output:
Log4
Log5
Only new logs are processed.
Real-World Problem: Log Rotation
Many systems rotate logs.
Example:
Before:
app.log
After rotation:
app.log.1
app.log
The old log is renamed:
app.log → app.log.1
A new empty:
app.log
is created.
Problem
Suppose:
Saved Offset = 50000
New log size:
200 Bytes
Offset is now invalid.
Solution
Check file size:
file_size = os.path.getsize("app.log")
if offset > file_size:
offset = 0
This resets reading to the beginning of the new file.
Production-Ready Example
import os
LOG_FILE = "app.log"
OFFSET_FILE = "offset.txt"
offset = 0
if os.path.exists(OFFSET_FILE):
with open(OFFSET_FILE, "r") as f:
offset = int(f.read())
file_size = os.path.getsize(LOG_FILE)
if offset > file_size:
offset = 0
with open(LOG_FILE, "r") as log:
log.seek(offset)
for line in log:
print(line.strip())
offset = log.tell()
with open(OFFSET_FILE, "w") as f:
f.write(str(offset))
Common Beginner Mistakes
Mistake 1: Not Saving Position
Bad:
with open("app.log") as log:
print(log.read())
Every execution:
Reads Entire File Again
Mistake 2: Forgetting tell()
Bad:
log.seek(offset)
but never updating offset.
Result:
Same Logs Read Repeatedly
Mistake 3: Not Handling Missing Offset File
Bad:
with open("offset.txt") as f:
Output:
FileNotFoundError
Always check:
os.path.exists()
Mistake 4: Ignoring Log Rotation
Bad:
log.seek(old_offset)
after log rotation.
Can miss logs or fail unexpectedly.
Mistake 5: Using read() on Huge Files
Bad:
content = log.read()
For a 5 GB log file:
High Memory Usage
Better:
for line in log:
print(line)
Best Practices
✅ Save offsets after every successful read
✅ Use tell() to track position
✅ Use seek() to resume reading
✅ Handle missing offset files
✅ Handle log rotation
✅ Read line by line for large logs
✅ Store offsets separately
✅ Use exception handling
Homework
Q1
Create:
app.log
Read it and display all contents.
Q2
After reading:
tell()
Display the current position.
Q3
Move to byte position:
10
using:
seek()
and read the remaining data.
Q4
Store the current position in:
offset.txt
Q5
Restart the program and continue reading from the saved position.
Q6
Modify the program to detect log rotation.
Homework Solutions
Solution 1
with open("app.log", "r") as log:
print(log.read())
Solution 2
with open("app.log", "r") as log:
print(log.tell())
Solution 3
with open("app.log", "r") as log:
log.seek(10)
print(log.read())
Solution 4
with open("app.log", "r") as log:
log.read()
position = log.tell()
with open("offset.txt", "w") as f:
f.write(str(position))
Solution 5
with open("offset.txt") as f:
offset = int(f.read())
with open("app.log") as log:
log.seek(offset)
print(log.read())
Solution 6
import os
file_size = os.path.getsize("app.log")
if offset > file_size:
offset = 0
Quick Revision
| Method | Purpose |
|---|---|
| open() | Open file |
| read() | Read content |
| readline() | Read one line |
| tell() | Current position |
| seek() | Move position |
| os.path.exists() | Check file exists |
| os.path.getsize() | Get file size |
| offset.txt | Save position |
Summary
To process only newly added log entries:
- Read the current log.
- Use
tell()to get the current position. - Save the position.
- Next run, use
seek()to return to that position. - Read only new entries.
- Update the saved position.
This technique is the foundation of:
- Log monitoring systems
- Log collectors
- Event processors
- Streaming data readers
- Production logging pipelines