Format y requires year 1900 on windows

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

ValueError due to Timestamp.strftime are currently silently caught by DatetimeLike array strftime and replaced with str(t). This leads to unexpected behaviour:

> dta = pd.DatetimeIndex(np.array(['1820-01-01', '2020-01-02'], 'datetime64[s]'))

> dta[0].strftime("%y")  # Instance operation raises ValueError
Traceback (most recent call last):
  (...)
  File "timestamps.pyx", line 1518, in pandas._libs.tslibs.timestamps.Timestamp.strftime
ValueError: format %y requires year >= 1900 on Windows

> dta.strftime("%y")  # Array operation catches error and turns into str representation (with default datetime format)
Index(['1820-01-01 00:00:00', '20'], dtype='object')

This very surprising behaviour on the array strftime is due to the try-except at

Note that this try-except is around since 0.16.1 (introduced by commit 3d54482)

This «questionable behaviour» was also reported in #48588

Note that this does not happen with dates that cannot be converted to datetime at all, because this particular ValueError is currently caught and turned into a NotImplementedError:

> dta = pd.DatetimeIndex(np.array(['-0020-01-01', '2020-01-02'], 'datetime64[s]'))

> dta.strftime("%Y_%m_%d")  # Custom so falls back on Timestamp.strftime
NotImplementedError: strftime not yet supported on Timestamps which are outside the range of Python's standard library. For now, please call the components you need (such as `.year` and `.month`) and construct your string from there.

> dta[0].strftime("%Y_%m_%d")
NotImplementedError: strftime not yet supported on Timestamps which are outside the range of Python's standard library. For now, please call the components you need (such as `.year` and `.month`) and construct your string from there.

Feature Description

I propose to add an errors parameter to array strftime with the following values

• ‘raise’ (default) would not catch any underlying error and raise them as is
• ‘ignore’ would catch all errors and silently replace the output with None instead of a string
• ‘warn’ would have the same behaviour as ‘ignore’ and would additionally issue a StrftimeErrorWarning warning message "The following timestamps could be converted to string: [...]. Set errors=‘raise’ to see the details"

> dta = pd.DatetimeIndex(np.array(['1820-01-01', '2020-01-02'], **'datetime64[s]'))**
> dta[0].strftime("%y")
ValueError: format %y requires year >= 1900 on Windows
> dta.strftime("%y")
ValueError: format %y requires year >= 1900 on Windows
> dta.strftime("%y", errors='raise')
ValueError: format %y requires year >= 1900 on Windows
> dta.strftime("%y", errors='ignore')
Index([None, '20'], dtype='object')
> dta.strftime("%y", errors='warn')
StrftimeErrorWarning : Not all timestamps could be converted to string: ['1820-01-01']. Set errors=‘raise’ to see the details
Index([None, '20'], dtype='object')

The specific NotImplementedError described previously can either disappear or stay, but in any case it should be handled the same way than ValueErrors above (meaning that if user selects ‘ignore’, the error must be silently caught)

Alternative Solutions

An alternative solution could be to always raise errors

Additional Context

No response

Python 3.5 suffers from a vulnerability caused by the behavior of the time_strftime() function. When called, the function loops over the format string provided, using strchr to search for each instance of '%'. After finding a '%', it continues to search two characters ahead, assuming that each instance is the beginning of a well formed format string token.

However, if a string ends with '%', this logic will result in a call to strchr that reads off the end of the format string buffer:

    /* check that the format string contains only valid directives */
    for(outbuf = strchr(fmt, '%'); <<<< Assuming fmt ends with a '%', this will return a pointer to the end of the string.
        outbuf != NULL;
        outbuf = strchr(outbuf+2, '%')) <<<< Once outbuf is pointing to the end of the string, outbuf+2 skips
    {                                        past the null terimnator, leading to a buffer over-read.
        if (outbuf[1]=='#')
            ++outbuf; /* not documented by python, */
        if ((outbuf[1] == 'y') && buf.tm_year < 0)
        {
            PyErr_SetString(PyExc_ValueError,
                        "format %y requires year >= 1900 on Windows");
            Py_DECREF(format);
            return NULL;
        }
    }

In some applications, it may be possible to exploit this behavior to disclose the contents of adjacent memory. The buffer over-read can be observed by running the following script:

from time import *
strftime("AA%"*0x10000)

Which, depending on the arrangement of memory, may produce an exception such as this:

0:000> g
(20b8.18d4): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=ffffffff ebx=52c1a6a0 ecx=00000000 edx=08ef3000 esi=08ec2fe8 edi=08ec2ff8
eip=52d254f3 esp=004cf9d4 ebp=004cfa58 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
python35!strchr+0x33:
52d254f3 f30f6f0a        movdqu  xmm1,xmmword ptr [edx] ds:002b:08ef3000=????????????????????????????????
0:000> db edx-0x10
08ef2ff0  41 25 41 41 25 41 41 25-00 d0 d0 d0 d0 d0 d0 d0  A%AA%AA%........
08ef3000  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3010  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3020  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3030  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3040  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3050  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
08ef3060  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
0:000> k5
ChildEBP RetAddr  
004cf9d0 52c1a7f6 python35!strchr+0x33 [f:\dd\vctools\crt\vcruntime\src\string\i386\strchr_sse.inc @ 75]
004cfa58 52c832d3 python35!time_strftime+0x156 [c:\build\cpython\modules\timemodule.c @ 615]
004cfa74 52ce442f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109]
004cfaa8 52ce18ec python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4651]
004cfb20 52ce339f python35!PyEval_EvalFrameEx+0x232c [c:\build\cpython\python\ceval.c @ 3184]

To fix this issue, it is recommended that time_strftime() be updated to check outputbuf[1] for null in the body of the format string directive validation loop. A proposed patch is attached.

Credit: John Leitch (johnleitch@outlook.com), Bryce Darling (darlingbryce@gmail.com)

#1 Фев. 13, 2011 12:21:22

datetime.datetime.date(date_fact).strftime("%d.%m.%Y")

year=1858 is before 1900; the datetime strftime() methods require year >= 1900

time.accept2dyear=0
time.strftime("%d.%m.%Y", datetime.datetime.date(date_fact).timetuple()),

year >= 1900 required