BulkCopy in mssql-python - API Spec & Discussion

[bewithgaurav]

Discussion for Bulk Copy in mssql-python (Issue #18)

We’re adding a BulkCopy API (cursor.bulkcopy()) to mssql-python for fast, high-throughput inserts into SQL Server.

Due to all the interest, this discussion thread is here for us to understand from you, about your usage scenarios - feel free to comment your suggestions and propose improvements!

If you’re using (or planning to use) bulk copy, feel free to comment with:

• Your use case (ETL, migration, backfill, analytics, etc.)
• Options you ended up using (batch_size, table_lock etc.)
• Anything that felt unclear or surprising

BulkCopy API Spec

The cursor.bulkcopy() method provides bulk data loading to SQL Server, similar to ADO.NET’s SqlBulkCopy.
It follows the standard DB-API cursor pattern and uses keyword arguments for configuration.

---

Method Signature

cursor.bulkcopy(
    table_name: str,
    data_source: Iterator[Tuple],
    **kwargs
) -> Dict[str, Any]

Full Signature with All Available Keyword Arguments

cursor.bulkcopy(
    table_name: str,
    data_source: Iterator[Tuple],

    # Performance settings
    batch_size: int = 1000,
    timeout_in_seconds: int = 30,

    # Column mapping
    column_mappings: Optional[List[Tuple[Union[str, int], str]]] = None,

    # Bulk copy options
    keep_identity: bool = False,
    check_constraints: bool = False,
    table_lock: bool = False,
    keep_nulls: bool = False,
    fire_triggers: bool = False,
    use_internal_transaction: bool = False
) -> Dict[str, Any]

---

Parameters

Required Positional Parameters

• table_name (str)
  Name of the destination table.

• data_source (Iterator[Tuple])
  Iterator yielding tuples of row data.

---

Keyword Arguments

All parameters beyond table_name and data_source must be passed as keyword arguments.

---

Performance Settings

• batch_size (int, default: 1000)
  Number of rows per batch.

  • Smaller values: lower memory usage, more network round trips
  • Larger values: higher memory usage, better throughput
  • Recommended range: 1000–10000

• timeout_in_seconds (int, default: 30)
  Operation timeout in seconds.

  • Set to 0 for no timeout
  • Increase for very large datasets

---

Column Mapping

• column_mappings (List[Tuple], optional)
  Maps source columns to destination columns.

  Format:

  [(source, destination), ...]

  • source: column name (str) or column index (int)
  • destination: destination column name (str)

  Example:

  [('src_id', 'UserID'), (0, 'FirstColumn')]

---

Bulk Copy Options

• keep_identity (bool, default: False)
  Preserve source identity values.
  Equivalent to SET IDENTITY_INSERT ON.

• check_constraints (bool, default: False)
  Enforces table constraints during bulk copy.
  Disabled by default for performance.

• table_lock (bool, default: False)
  Acquires a table-level lock to reduce locking overhead.
  Recommended when exclusive access is available.

• keep_nulls (bool, default: False)
  Preserves explicit NULL values instead of applying column defaults.

• fire_triggers (bool, default: False)
  Fires INSERT triggers during bulk copy.
  Disabled by default for performance.

• use_internal_transaction (bool, default: False)
  Wraps the bulk copy operation in an internal transaction.
  Automatically rolls back on error.

---

Return Value

Returns a dictionary containing operation statistics:

{
    'rows_copied': 10000,       # Number of rows successfully copied
    'batch_count': 10,          # Number of batches processed
    'elapsed_time': 1.5,        # Total elapsed time (seconds)
    'rows_per_second': 6666     # Throughput
}

Looking forward to your feedback!

cc: @dlevy-msft-sql @saurabh500 @sumitmsft

[saurabh500]

Re

Column Mapping
...
column_mappings (List[Tuple], optional)
source: column name (str) or column index (int)

I think we dont have a usecase for column name (str) in the source.
We may end up limiting the source to only support column index(int)

[amachanic]

Agree, column name in the source is not useful, unless you decide to support mapping collections there - which I also don't think is a necessary thing.

What I would ideally like to see here personally is a slightly simpler implementation: Just let the user pass a list of destination column names, in order, that correspond to the input tuples. So if my input tuple has three elements corresponding to colB, colA, and colC respectively, I'd like to pass in ['colB', 'colA', 'colC'], rather than [(0, 'colB'), (1, 'colA'), (2, 'colC')].

[saurabh500]

I like this suggestion. We will consider this an addition or replacement over the proposed column mapping APIs.
I will take this topic up with the team internally.

[amachanic]

I'd like to understand the reasoning behind hiding so much behind the kwargs instead of using optional parameters. I feel that this really makes the interface much more difficult to use. (The various options aside from the column mappings -- which I feel should be a first-class thing -- could all be grouped in a named tuple or class with defaults, if you want to cut it down to just two optional params.)

[saurabh500]

Thanks, I appreciate this suggestion and I can see the benefit you are alluding to. We will make an update to the proposed API and ping this discussion thread.

cc @bewithgaurav

[jblair-LCOG]

For clarity, are we talking about requiring a **kwargs to get at those, or that passing nondefaults for those arguments requires the keyword (i.e. those come after a * in the signature)?

Most of the extra args are booleans, so allowing them as positional args fall into the Flake8 boolean trap. Requiring the keyword for less common args makes a pretty clean interface.
I tend to agree that column_mappings will be used in most calls, so probably deserves to be promoted from keyword-land.

[amachanic]

**kwargs. (See the cursor.bulkcopy spec above.) The * feature is a good solution for this, but seems to be unpopular for some reason. Optional params with defaults are fine too IMO. If users decide to do crazy things with positional args I think that's their problem, not that of the library!

[bewithgaurav]

@amachanic - Thanks for the suggestion!

Completely agreed on both points - we can let users take care of param positioning (library shouldn't necessarily enforce with *), and the interface should expose explicit params rather than hiding everything behind **kwargs.

We'll revise the API spec and will ping this thread once updated.

[leonkosak]

My use case is for ELT for fast inserts from parquet file. I agree with @amachanic about hiding so much parameters behind **kwargs.
What performance is expected compared to OPENROWSET command where SQL engine directly reads parquet files?